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Resources  Management  Guide,  3)  Test  and  Evaluation  Management  Guide,  4)  Risk 
Management  Concepts  and  Guidance,  5)  DoD  Manufacturing  Manag  :ment  Handbook,  and 
6)  Subcontracting  Management  Handbook,  'j  f  //M  j,  k  N 
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' — This  Systems  Engineering  Management  Guide  {SEMG)  is  designed  to  A)  acquaint  the 
newcomer  with  systems  engineering  concepts  and  techniques  and  2)  identify  relevant  directives 
and  references.  These  concepts,  when  combined  with  common  sense  and  technical  expertise, 
constitute  the  basis  of  a  sound  systems  engineering  program.  The~i3EMG -highlights  the 
technical  management  activities  over  the  system’s  life  cycle  from  program  initiation  to  system 
disposal.  All  activity  centers  around  the  system  itself;  thus,  the  system  configuration  at  any  time 
is  of  common  interest  to  all  engineering  disciplines.  These  activities  are  normally  divided  into 
functional  areas  of  design,  test,  manufacturing,  and  logistics  support.  Each  of  these  functional 
ar«K  is  active  throughout  the  system's  life  cycle.  -  '  VZM'  U  , 

The  effort  involved  in  the  acquisition  process  dm  be  modeled  as  an  input,  process,  and 
output.  The  input  is  the  need  and  constraints  provided  by  the  user.  The  process  consists  of 
managing  the  technical  activities  by  establishing  and  maintaining  a  balance  among  system 
effectiveness,  schedule,  and  cost.  This  is  accomplished  through  systems  engineering.  The  output  / 
is  the  system  itself.  The  goal  of  the  acquisition  process,  therefore,  is  to  deploy,  in  a  timely  7 
manner,  and  sustain  an  effective  system  that  satisfies  a  specific  user’s  need  at  an  affordable 
cost. 

To  summarize,  management  of  the  acquisition  process  can  be  define^Las  the  logical  and 
systematic  effort  required  to  transform  a  military  need  jntg_  an-operational  system  which 
requires  a  cooperative  effort  on  the  part  of  jjoycrament  and  industry.  The  capability  of  the 
industrial  base  to  economically  produce  Defense  systems,  on  a  timely  basis,  is  a  key  element 
of  the  acquisition  process. 
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CHAPTER  1 


SYSTEMS  ENGINEERING  OVERVIEW 


1.1  INTRODUCTION 

The  objective  of  this  guide  is  to 
familiarize  you  with  the  role  that  systems 
engineering  plays  in  the  development  of  a 
weapon  system.  It  will  focus  primarily  on 
the  relationship  between  the  technical 
management  process  and  the  systems 
engineering  process.  The  guide  is  based  on 
the  tasks  defined  in  MIL-STD-499A  [1],  and 
the  concepts  and  processes  defined  in 
"Systems  Engineering  and  Analysis",  by 
Benjamin  S.  Blanchard  [2J.  The  guide  is 
intended  to  provide  the  perspective  and 
background  data  in  systems  engineering 
necessary  for  effective  overall  program 
management.  It  relates  the  diverse  elements 
of  systems  engineering  not  only  to  each 
other,  but  to  overall  system  effectiveness  in 
satisfying  a  defined  user  need  at  an 
affordable  cost. 

1.2  EVOLUTION  OF  SYSTEMS 
ENGINEERING 

The  past  several  decades  have  seen  the 
rise  of  large,  highly  interactive  systems  that 
are  on  the  forward  edge  of  technology.  As  a 
result  of  this  growth  and  the  increased  usage 
of  digital  systems  (computers  and  software), 
the  concept  of  systems  engineering  has 
gained  increasing  attention.  Some  of  this 
attention  is  no  doubt  due  to  large  program 
failures  which  possibly  could  have  been 


avoided,  or  at  least  mitigated,  through  the 
use  of  systems  engineering  principles.  The 
complexity  of  modem  day  weapon  systems 
requires  conscious  application  of  systems 
engineering  concepts  to  ensure  producible, 
operable,  and  supportable  systems  that  satisfy 
mission  requirements. 

Although  many  authors  have  traced  the 
roots  of  systems  engineering  to  earlier  dates, 
the  initial  formalization  of  the  systems 
engineering  process  for  military  development 
began  to  surface  in  the  mid-1950s  on  the 
ballistic  missile  programs.  These  early  ballistic 
missile  development  programs  marked  the 
emergence  of  engineering  discipline 
"specialists"  which  has  since  continued  to 
grow.  Each  of  these  specialties  not  only  has 
a  need  to  take  data  from  the  overall 
development  process,  but  also  to  supply  data, 
in  the  form  of  requirements  and  analysis 
results,  to  the  process. 

A  number  of  technical  instructions, 
military  standards  and  specifications,  and 
manuals  were  developed  as  a  result  of  these 
development  programs.  In  particular,  MIL- 
STD-499  was  issued  in  1969  to  assist  both 
government  and  contractor  personnel  in 
defining  the  systems  engineering  effort  in 
support  of  defense  acquisition  programs.  This 
standard  was  updated  to  MIL-STD-499A  [1] 
in  1974,  and  formed  the  foundation  for 
current  application  of  systems  engineering 
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principles  to  military  development  programs. 

13  DEFINITIONS  OF  SYSTEMS 

ENGINEERING 

MIL-STD-499A  [1]  defines  systems 
engineering  as: 

"  the  application  of  scientific  and 
engineering  efforts  to  (a)  transform  an 
operational  need  into  a  description  of  system 
performance  parameters  and  a  system 
configuration  through  the  use  of  an  il  rative 
process  of  definition,  synthesis,  analysis,  design, 
test,  and  evaluation;  (b)  integrate  related 
technical  parameters  and  ensure  compatibility 
of  all  physical,  functional,  and  program 
interfaces  in  a  manner  that  optimizes  the  total 
system  definition  and  design;  (c)  integrate 
reliability,  maintainability,  safety,  survivability, 
human  engineering,  and  other  such  factors  into 
the  total  engineering  effort  to  meet  cost, 
schedule,  support  ability,  and  technical 
performance  objectives 

In  its  simplest  terms,  systems  engineering 
is  both  a  technical  process  and  a 
management  process.  The  above  definition 
focuses  in  on  the  technical  aspects.  To 
successfully  complete  the  development  of  « 
system,  both  aspects  must  be  applieu 
throughout  the  system  life  cycle.  From  a 
government’s  program  management  point  of 
view,  DSMC  favors  the  management 
approach  and  defines  systems  engineering  as 
follows: 

" Systems  engineering  is  the  management 
function  which  controls  the  total  system 
development  effort  for  the  purpose  of  achieving 
an  optimum  bqfoppr  pf  till  system  elements .  It 


is  a  process  which  transforms  an  operational 
need  into  a  description  of  system  parameters 
and  integrates  those  parameters  to  optimize  the 
overall  system  effectiveness 

A  system  life  cycle  begins  with  the 
user’s  needs,  expressed  as  constraints,  and 
the  capability  requirements  needed  to  satisfy 
mission  objectives.  Systems  engineering  is 
essential  in  the  earliest  planning  period,  in 
conceiving  the  system  concept  and  defining 
system  requirements.  As  the  detailed  design 
is  being  done,  systems  engineers  assure:  1) 
balanced  influence  of  all  required  design 
specialties,  2)  resolve  interface  problems,  3) 
conduct  design  reviews,  4)  perform  trade-off 
analyses,  and  5)  assist  in  verifying  system 
performance.  During  the  Production  phase, 
systems  engineering  is  concerned  with:  1) 
verifying  system  capability,  2)  maintaining 
the  system  baseline,  and  3)  forming  an 
analytical  framework  for  producibility 
analysis.  During  the  Operation  and  Support 
(O/S)  phase,  systems  engineering:  1) 
evaluates  proposed  changes  to  the  systems, 
2)  establishes  their  effectiveness,  and  3) 
facilitates  the  effective  incorporation  of 
changes,  modifications,  and  updates. 

1.4  THE  SYSTEMS  ENGINEERING 
PROCESS 

Although  programs  differ  in  underlying 
requirements,  there  is  a  consistent,  logical 
process  for  best  accomplishing  system  design 
tasks.  Figure  1-1  illustrates  the  activities  of 
the  basic  systems  engineering  process.  This 
process  is  described  in  detail  in  Chapters  5 
through  8  of  this  guide. 

The  systems  engineering  process  is 
iteratively  applied.  It  consists  primarily  of 
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four  activities:  1)  functional  analysis,  2) 
synthesis,  3)  evaluation  and  decision,  and  4) 
a  description  of  systems  elements.  The 
product  element  descriptions  become  more 
detailed  with  each  application  and  support 
the  subsequent  systems  engineering  design 
cycle.  The  final  product  is  production-ready 
documentation  of  all  system  elements. 

1.S  SYSTEMS  ENGINEERING 
OBJECTIVES 

Since  the  requirement  to  implement  a 
systems  engineering  process  may  cause  major 
budgetary  commitments  and  impact  upfront 
development  schedules,  it  is  important  to 
understand  the  inherent  objectives: 

a.  Ensure  that  system  definition  and 
design  reflects  requirements  for  fiU  system 
elements:  equipment,  software,  personnel, 
facilities,  and  data. 


b.  Integrate  technical  efforts  of  the 
design  team  specialists  to  produce  an 
optimally  balanced  design. 

c.  Provide  a  comprehensive  indentured 
framework  of  system  requirements  for  use  as 
performance,  design,  interface,  support, 
production,  and  test  criteria. 

d.  Provide  source  data  for  development 
of  technical  plans  and  contract  work 
statements. 

e.  Provide  a  systems  framework  for 
logistic  analysis,  integrated  logistic  support 
(ILS)  trade  studies,  and  logistic 
documentation. 

f.  Provide  a  systems  framework  for 
production  engineering  analysis,  producibility 
trade  studies,  and  production/manufacturing 
documentation. 


Figure  1-1 

The  Systems  Engineering  Process 
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g.  Ensure  that  life  cycle  cost 
considerations  and  requirements  are  fully 
considered  in  all  phases  of  the  design 
process. 

1.6  SYSTEMS  ENGINEERING 

IMPLEMENTATION 

Successful  application  of  systems 

engineering  requires: 

a.  Mutual  understanding  and  support 

between  the  military  and  contractor  Program 
Managers.  They  must  be  willing  to  make 
the  systems  engineering  process  the 

backbone  of  the  overall  development 

program. 

b.  Understanding  the  need  to  define  and 

communicate  among  the  engineering 

specialty  programs. 

c.  Recognition  of  the  role  of  formal 
technical  reviews  and  audits,  as  described  in 
MIL-STD-1521B  [3],  including  the  value, 
objectives,  and  uniqueness  of  each  formal 
review  and  audit. 

d.  Knowledge  of  the  objectives  of  the 
program. 

e.  A  thorough  interpretation  of  the  user’s 
requirements. 

1.7  SYSTEMS  ENGINEERING  OUTPUTS 

The  output  of  the  systems  engineering 
process  is  documentation.  This  is  the  means 
by  which  it  controls  the  evolutionary 
development  of  the  system.  Systems 
engineering  prepares  a  number  of  technical 
management  and  engineering  specialty  plans 


which  define  how  each  phase  of  the 
acquisition  cycle  will  be  conducted.  Draft 
plans  are  usually  submitted  with  the  proposal 
and  final  plans  are  delivered  in  accordance 
with  the  Contract  Data  Requirements  List 
(CDRL).  These  plans  are  used  by  the 
government  to  ensure  compliance  with  the 
contract  and  used  by  the  contractor  to 
develop  detailed  schedules  and  allocation  of 
resources.  Specifications  are  submitted  which 
form  the  basis  for  the  design  and 
development  effort.  Top  level  specifications 
are  incorporated  into  the  statement  of  work 
(SOW)  and  provided  to  the  developer.  The 
developer  will  allocate  these  top  level 
requirements  to  lower  level  system 
components  (hardware  and  software)  and 
submit  the  associated  specifications  and 
design  documents  to  the  government  for 
approval.  The  status  of  system  development 
progress  is  tracked  and  documented  in  the 
form  of  technical  review  data  packages, 
technical  performance  measurement  (TPM) 
reports,  analysis  and  simulation  reports,  and 
other  technical  documentation  pertinent  to 
the  program.  In  summary,  this 
documentation  may  include: 

a.  Systems  Engineering  Management 
Plan  (SEMP) 

b.  Specifications  (system,  segment, 
development,  product,  process,  material,) 

c.  Design  Documentation 

d.  Interface  Control  Documents  (ICDs) 

e.  Risk  Analysis  Management  Plan 

f.  Survivability/  Vulnerability  (S/V) 
Hardness  Plan 
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g.  Mission  Analysis  Report 

h.  Reliability  Plan 

i.  Maintainability  Plan 

j.  Integrated  Logistics  Support  Plan 
(ILSP) 

k.  Software  Development  Plan  (SDP) 

l.  Test  and  Evaluation  Master  Plan 
(TEMP) 

m.  Producibility  Plan 
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CHAPTER  2 


SYSTEMS  ENGINEERING  IN  THE 
ACQUISITION  PROCESS 


2.1  INTRODUCTION 

Throughout  any  design  process,  systems 
engineering  is  used  to  identify  and  define  the 
functional  characteristics  of  system  hardware, 
software,  facilities,  data,  and  personnel.  It  is 
an  interactive  process  of  analysis  and  design, 
with  the  objective  of  satisfying  an  operational 
mission  need  in  the  most  cost-effective 
manner.  The  systems  engineering  process  is 
used  to  analyze  mission  requirements  and 
translate  them  into  design  requirements  at 
successively  lower  levels. 

This  chapter  discusses  systems  engineering 
in  the  context  of  the  Department  of  Defense 
(DoD)  acquisition  process.  A  description  of 
government  acquisition  policies  and  the 
relationship  of  the  systems  engineering 
process  with  the  life  cycle  of  a  typical  system 
are  included. 

22  GOVERNMENT  ACQUISITION 
POLICIES 

The  Department  of  Defense  uses  a 
systematic  technical  management  process  to 
control  acquisition  programs,  as  illustrated  in 
Figure  2-1.  The  DoD  acquisition  process  for 
major  systems  has  its  foundation  in  federal 
policy. 

In  the  early  1970s,  the  Office  of  Federal 
Procurement  Policy  (OFPP)  was  established 


to  provide  policies,  methods,  and  criteria  for 
the  acquisition  of  property  and  services  for 
all  executive  agencies.  In  1976,  the  Office  of 
Management  and  Budget  (OMB)  Circular 
A- 109  was  published  with  the  goal  of 
increasing  management  effectiveness  on 
major  system  acquisitions.  The  circular  laid 
the  foundation  for  standardizing  the 
government  acquisition  process  and 
promoting  unbiased  concept  definition. 
OMB  Circular  A-109  requires  the 
government  operating  agency  to  establish  and 
justify  a  valid  requirement  for  a  capability, 
which  must  be  approved  by  the  executive 
agency  head  (e.g.;  Secretary  of  Defense 
(SECDEF),  NASA  Administrator)  before 
involving  industry  in  the  system  acquisition 
process. 

The  principal  guidance  for  defense  system 
acquisitions  is  DoDD  5000.1,  "Major  and 
Non-Major  Defense  Acquisition  Programs" 
[1]  and  DoDI  5000.2,  "Defense  Acquisition 
Program  Procedures"  [2J.  These  documents 
reflects  certain  acquisition  management 
principles  and  objectives: 

a.  Ensure  effective  design  and  price 
competition 

b.  Improve  system  readiness  and 
sustainability 

c.  Increase  program  stability  through 
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effective  long-range  planning,  use  of 
evolutionary  alternatives,  realistic  budgeting 
and  funding  of  programs  for  the  total  life 
cycle,  and  planning  to  achieve  economical 
production  rates 

d.  Delegate  authority  to  the  lowest 
levels  of  the  service  that  can  provide  a 
comprehensive  review  of  the  program 

e.  Achieve  a  cost-effective  balance 
between  acquisition  costs,  ownership  costs, 
and  system  effectiveness  in  terms  of  the 
missions  to  be  performed. 

2J  SYSTEM  LIFE  CYCLE 

The  acquisition  process  for  major  defense 
systems,  defined  in  DoDD  5000.1  and  DoDI 
5000.2,  is  depicted  in  Figure  2-1  and  consists 
of  five  primary  phases:  Concept  Exploration/ 
Definition  (C/E),  Concept  Demonstration/ 
Validation  (D/V),  Full  Scale  Development 
(rSD),  Production  and  Deployment  (P/D), 
and  Operation  and  Support  (O/o).  Within 
DoD,  major  systems  are  defined  as  systems 
anticipating  funding  levels  of  more  than  $200 
million  in  research,  development,  test,  and 
evaluation  or  $1  billion  in  procurement  in 
constant  FY-80  (fiscal  year  1980)  dollars. 
The  process  begins  by  conducting  mission 
area  analysis  efforts  in  determining  the  user’s 
mission  need,  followed  by  an  approval  for 
program  initiation  and  authority  to  budget  for 
a  new  program  (Milestone  0  decision).  Next, 
the  program  normally  enters  the  C/E  phase 
during  which  all  reasonable  system 
alternatives  are  explored.  The  program  then 
enters  the  D/V  phase  (Milestone  I  decision) 
during  which  the  preferred  system  concepts 
are  selected  and  tested  to  identify  risk  areas 
and  to  demonstrate  that  all  experimental 


efforts  have  been  completed.  The  results  of 
these  efforts  are  reviewed  and  the  program 
is  allowed  to  enter  into  the  FSD  phase 
(Milestone  II  decision)  where  system  detailed 
design  and  test  are  performed.  Once 
completed,  the  system  is  approved  to  proceed 
with  full-rate  production  and  initial 
deployment  in  the  P/D  phase  (Milestone  III 
decision)  during  which  the  actual  production 
or  construction  takes  place.  The  initial 
deployment  also  marks  the  beginning  of  the 
O/S  phase.  A  review  will  be  conducted  one 
to  two  years  after  initial  deployment  to 
assure  that  the  operational  readiness  and 
support  objectives  are  being  achieved 
(Milestone  IV  decision).  Later,  during  the 
O/S  phase,  modifications  and  product 
improvements  are  usually  implemented. 
Another  review  will  be  conducted  somewhere 
between  five  to  ten  years  after  initial 
deployment  (Milestone  V  decision)  to 
determine  if  major  upgrades  are  needed. 

At  the  end  of  each  phase,  as  noted 
above,  the  need  for  the  program  is  re¬ 
certified,  usiag  milestone  decision  reviews,  by 
the  Secretary  of  Defense  (SECDEF)  or  the 
Service  Secretary,  as  required,  before 
additional  resources  are  authorized.  At  each 
review,  the  decision  authority  can  choose  to 
continue  the  present  phase,  proceed  to  the 
next  phase,  or  cancel  the  program.  The 
SECDEF  can  also  direct  a  DoD  program  to 
omit  C/E  and/or  D/V  and  proceed  directly 
into  FSD  as  special  circumstances  prevail 

The  SECDEF  is  assisted  in  this 
decision-making  role  by  the  Defense 
Acquisition  Board  (DAB),  chaired  by  the 
Under  Secretary  of  Defense  for  Acquisition 
(USD(A)).  The  DAB  has  broad  review 
responsibility  for  Milestones  I  through  V; 
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Figure  2-1 

Acquisition  Process  for  Major  Defense  Systems 


IF  NOT  DESIGNATED  “MAJOR,*'  000  COMPONENT  RETAINS 
DECISION  RESPONSIBILITY  APPLYING  MANAGEMENT 
PRINCIPLES  ANO  OBJECTIVES  OF  DOQ  DIRECTIVE  SOOO.I 


aids  SECDEF  in  defining  and  validating  new 
system  requirements;  examines  trade-offs 
between  cost  and  performance;  explores 
alternatives  to  new  research  and  development 
starts;  and  recommends  full  scale 
development  and  full  rate  production. 

The  following  discussion  focuses  on  the 
acquisition  process  for  major  defense  systems. 
A  modified  approach,  unique  to  each  service, 
is  employed  for  non-major  or 
non-developmental  systems.  Although  the 
approval  authority  level  for  the  non-major 
systems  is  not  as  high  as  the  USD(A),  the 
aspects  of  the  program  that  must  be 
demonstrated  are  identical.  Non-major 
systems  may  not  follow  all  of  the  acquisition 
phases  that  major  systems  do.  They  will, 
however,  have  a  development  and  review 
cycle  appropriate  to  the  nature  and  scope  of 
the  program  in  accordance  with  the 
streamlining  process  outlined  in  Chapter  10 
of  this  guide.  Acquisition  of  non-major 
systems  is  directed  by  the  service  or  major 
command,  as  appropriate.  Each  service  has 
its  own  system  of  reviews  and  approvals  that 
achieves  objectives  similar  to  the  DoD  cycle. 

2-3.1  Program  Initiation  Mission  Need 
Decision 

Mission  area  analysis  (MAA)  is  an 
ongoing  activity  for  identifying  deficiencies  in 
existing  defense  capabilities  or  determining 
more  effective  means  of  performing  assigned 
tasks  within  assigned  mission  areas.  When 
deficiencies  or  opportunities  may  be 
identified,  system  performance  requirements 
are  established.  MAA  takes  cognizance  of 
changes  in  national  defense  policy,  external 
threats,  and  technological  capabilities.  The 
analysis  considers  alternatives  to  new 


development  such  as  redeployment  of  existing 
military  resources,  use  of  commercial  systems, 
or  tactics  changes.  When  no  other 
alternative  is  available,  the  product  of  this 
activity  is  development  of  the  Mission  Need 
Statement  (MNS).  The  MNS  defines  the 
mission  need,  identifies  constraints,  and 
outlines  the  initial  acquisition  strategy. 

2  J 2  Concept  Exploration/  Definition  Phase 

C/E  is  initiated  following  the  DAB 
approval  of  the  MNS  and  the  issuance  of 
the  Acquisition  Decision  Memorandum 
(ADM).  The  USD(A)  forwards  the  ADM  to 
the  SECDEF  for  his  endorsement  of  this 
proposed  new  start  and  designation  as  a 
major  system.  The  MNS  is  normally  included 
in  the  service  Program  Objectives 
Memorandum  (POM). 

During  C/E,  system  concepts  are 
defined  and  selected  for  further  development 
The  systems  engineer,  working  through 
private  industry  and  DoD  research  and 
development  (R&D)  agencies,  identifies  all 
reasonable  system  alternatives  that  may 
satisfy  the  mission  need  and  makes 
recommendations  to  the  program  office.  The 
Program  Manager  then  selects  those 
alternatives  or  concepts  which  meet  cost,  risk, 
schedule,  and  readiness  objectives. 

Alternative  system  design  concepts  are 
explored  through  competitive,  parallel, 
short-term  contracts;  alternative  methods  of 
logistic  support  are  examined  through  logistic 
support  analysis;  and  producibility  is  analyzed 
through  producibility  engineering  and 
planning.  Contractors  are  provided  with 
operational  employment  intentions,  mission 
performance  criteria,  and  life  cycle  cost 
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(LCC)  estimating  factors.  Affordability  is 
assessed  and  early  gross  LCC  estimates  of 
the  competing  alternatives  are  made.  Design 
to  cost  (DTC)  constraints  are  used  to 
indictate  which  producibility  and  production 
efficiencies  are  required. 

The  industry’s  systems  engineering  activity 
during  this  period  is  based  on  system 
requirements  provided  with  the  statement  of 
work  (SOW).  These  requirements  are 
translated  into  alternative  design  concepts, 
through  functional  analysis,  synthesis,  and 
trade-off  analysis.  For  each  segment  of  the 
design  concept,  allocated  requirements, 
interface  identifications,  and  technical  budgets 
are  produced  as  systems  engineering 
products.  The  industry  output  is  reviewed  by 
the  government  for: 

a.  Capability  of  the  proposed  system  to 
meet  the  mission  need  and  program 
objectives,  including  resources  required  and 
associated  risk  levels 

b.  Benefits  to  be  derived  by  trade-offs 
among  technical  performance,  operational 
effectiveness  and  suitability  (system 
effectiveness),  LCC  (including  age  and 
operation),  and  schedule  (time  to  develop, 
manufacture,  and  field) 

c.  Relevant  development  experience  and 
performance  record  of  each  competitor  and 
the  competence  of  competitors’  key  systems 
engineering  and  design  personnel. 

System  descriptions,  and  associated  risk, 
cost,  and  development  time  estimates  are 
used  by  the  government  to  establish  a  system 
functional  baseline,  usually  in  the  form  of  a 
Type  A  system  specification  (refer  to 


M1L-STD-490A  [3]).  This  baseline  should 
not  constitute  selection  of  a  specific  design 
concept,  but  rather  identification  of  feasible, 
affordable  ranges  of  cost  and  system 
effectiveness.  Proper  identification  is 
essential  to  an  effective  acquisition  strategy 
since  real  competition  requires  a  system  level 
specification  which  can  be  met  by  more  than 
one  design  concept. 

Systems  Engineering  Management  Plans 
(SEMPs),  Integrated  Logistic  Support  Plans 
(ILSPs),  Computer  Resources  Life  Cycle 
Management  Plans  (CRLCMPs),  Test  and 
Evaluation  Master  Plans  (TEMPs),  and  other 
functional  plans  are  normally  initiated  during 
this  phase.  A  System  Requirements  Review 
(SRR)  is  accomplished  to  determine  the 
extent  to  which  selected  contractor  design 
concepts  satisfy  the  stated  mission  need. 

In  order  to  support  the  Milestone  I 
decision,  a  System  Concept  Paper  (SCP)  is 
prepared  to  summarize  the  results  of  the 
C/E  phase;  an  initial  Test  and  Evaluation 
Master  Plan  (TEMP)  is  developed  to  address 
program  test  requirements;  and  other 
documentation  is  prepared  to  establish  the 
program  charter  and  to  refine  the  program 
acquisition  strategy  as  necessary. 

The  SCP  for  a  major  system  is  reviewed 
first  by  the  service  component’s  System 
Acquisition  Review  Council  (SSARC),  and  if 
approved,  by  the  DAB.  The  DAB  review  at 
Milestone  I  reconfirms  the  program  need, 
determines  that  program  risks  were 
adequately  considered,  and  ensures  that 
adequate  planning  for  technical  performance, 
supportability,  test  and  evaluation, 
producibility,  and  life  cycle  costing  was 
established.  When  the  SCP  meets  all  of 
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these  objectives*  it  is  forwarded  to  the 
SECDEF  with  recommendations  to  proceed 
to  D/V  or  FSD.  Approval  by  the  SECDEF 
is  documented  in  the  ADM  and  authorizes 
the  service  to  prepare  and  release  a  request 
for  proposal  (RFP)  for  the  approved  phase. 

2JJ  Concept  Demonstration/Validation 
Phase 

The  D/V  phase  is  normally  initiated  by 
the  release  of  the  ADM.  The  D/V  phase 
RFP  may  be  released  prior  to  the  ADM; 
however,  contract  award  must  follow  the 
ADM.  The  D/V  phase  RFP  contains  a 
system  level  specification,  the  program 
management  approach,  and  the  SOW 
describing  the  scope  of  the  contractor  effort. 
After  proposal  evaluation  and  contract 
award,  systems  engineering  becomes  a 
contractor  effort,  often  by  two  or  more 
contractors.  The  government  usually  assumes 
a  systems  engineering  management  role. 

The  objective  in  the  D/V  phase  is  to 
identify  and  analyze  major  system 
alternatives,  to  examine  risky  subsystems,  and 
to  determine  whether  or  not  to  proceed  into 
FSD.  The  main  products  of  this  phase  are 
normally  a  validated  system  specification 
(Type  A)  which  determines  the  system 
functional  baseline  and  an  initial  set  of 
subsystem  performance  development 
specifications.  When  validated,  these 
development  specifications  (Type  B)  will 
determine  the  allocated  baseline  for  the 
system.  The  allocated  baseline  is  also 
referred  to  as  the  "design  requirements"  or 
the  "design  to"  baseline.  It  incorporates 
technological  approaches  developed  to  satisfy 
requirements  established  at  the  system  level 
by  the  functional  baseline. 


Another  major  product  of  the  D/V  phase 
is  the  Systems  Engineering  Management  Plan 
(SEMP),  which  includes  plans  for  risk 
alleviation  and  identifies  the  schedule  for 
producing  all  required  plans  for  the 
supporting  engineering  specialties,  such  as 
electromagnetic  compatibility/  electromagnetic 
interference  (EMC/  EMI),  safety,  reliability, 
maintainability,  integrated  logistic  support, 
and  human  engineering.  MIL-STD-499A  [4] 
details  the  information  to  be  included  in  the 
SEMP  and  suggests  that  it  be  tailored  to  suit 
contractor  requirements.  Chapter  3  of  this 
guide  addresses  the  considerations  of  SEMP 
development,  timing,  and  format.  Other 
products  include  updated  ILSPs,  CRLCMPs, 
and  TEMPs. 

As  the  systems  engineering  process 
progresses  from  the  functional  to  the 
allocated  baselines,  required  configuration 
items  (CIs)  are  identified.  The  process 
includes  trade  off  analyses  to  ensure  that  the 
system  will  satisfy  the  functional  baseline  with 
the  optimal  balance  of  LCC,  schedule,  and 
system  effectiveness.  Logistic  support 
analyses  are  conducted  to  identify  and 
analyse  logistic  support  alternatives  for  the 
system.  The  selected  support  alternative  is 
documented  in  the  ILSP. 

Elements  of  the  proposed  system  are 
continually  assessed  to  identify  areas  of 
technical  uncertainty  that  must  be  resolved 
in  later  program  phases  (risk  assessment). 
Critical  components  may  be  prototyped  to 
reduce  risk.  A  System  Design  Review  (SDR) 
is  held  at  the  end  of  the  D/V  phase  (or  early 
in  the  FSD  phase)  to  review  the  preliminary 
allocation  of  requirements  tc  hardware  CIs 
(HWCI),  computer  software  CIs  (CSCI), 
personnel,  facilities,  and  data. 
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A  Decision  Coordinating  Paper  (DCP)  is 
prepared  by  the  SSARC(s)  for  review  by  the 
DAB.  If  all  requirements  are  satisfied,  a 
ratified  ADM  recommending  FSD  approval 
is  forwarded  to  the  SECDEF.  Once 
approved,  the  ADM  authorizes  the  service  to 
prepare  and  release  an  RFP  for  FSD 
activities. 

23.4  Full  Scale  Development  Phase 

To  initiate  the  FSD  phase,  the 
government  selects  the  best  proposal(s)  and 
negotiates  a  development  contract  with  the 
contractor(s).  More  than  one  contractor  may 
be  retained  through  FSD  to  maintain  the 
competitive  environment  of  D/V,  reduce 
program  risk,  or  provide  multiple  production 
sources,  as  the  acquisition  strategy  dictates. 
The  purpose  of  the  FSD  phase  is  to  provide 
the  design  documentation  necessary  to  go  to 
rate  production  and  the  1LS  documentation 
necessary  to  field  and  fully  support  the 
system.  This  is  done  by  completing  detailed 
design,  and  by  demonstrating  that  reliability, 
producibility,  supportability,  testability,  and 
performance  requirements  have  been  met. 
Continual  assessment  of  risk  using  technical 
performance  measurement  (TPM)  and  cost 
schedule  control  system  criteria  (CSCSC)  is 
also  characteristic  of  this  phase.  The  FSD 
design  activity  is  based  on  the  development 
specifications  (Type  B)  and  systems 
engineering  documentation,  with  such 
changes  as  may  result  from  a  ratified  DCP. 

The  SEMP  is  implemented  at  the 
beginning  of  the  FSD  phase,  if  not  during  the 
D/V  phase.  Detailed  system  simulations  may 
be  developed  to  predict  system  performance 
and  establish  specific  performance 
parameters.  Plans  developed  in  the  previous 


phases  are  implemented.  Test  plans  are 
developed,  tests  are  conducted,  and  test  data 
are  audited  and  compiled. 

A  Preliminary  Design  Review  (PDR)  is 
conducted  for  each  Cl  and  normally  follows 
the  SDR  prior  to  the  start  of  detailed  design. 

It  provides  authentication  of  the  development 
specifications  (Type  B)  and  establishes  the 
allocated  baseline  for  CIs.  For  hardware, 
the  allocated  baseline  for  HWCIs  is  normally 
established  at  PDR,  but  no  later  than  Critical 
Design  Review  (CDR).  For  software, 
however,  the  Software  Specification  Review 
(SSR)  provides  authentication  of  the  software 
development  specifications  (Type  B-5)  and 
establishes  the  allocated  baseline  for  CSCIs. 

The  SSR  also  follows  the  SDR,  but  may 
occur  before  or  after  the  hardware  PDR 
depending  on  the  amount  of  effort  required 
to  authenticate  the  software  specifications.  In 
any  case,  the  SSR  will  be  conducted  prior  to 
the  software  PDR. 

I 

i 

A  Critical  Design  Review  (CDR)  is 
conducted  for  each  Cl  before  the  design  is 
released  for  production  as  a  developmental 
item.  Systems  engineering  activities  change 
considerably  in  nature  after  CDR  and  consist 
primarily  of  resolving  interface  compatibility 
problems  and  solving  technical  problems 
discovered  during  development  testing. 
Following  the  CDR,  a  Test  Readiness  Review 
(TRR)  is  held  for  CSCIs  to  review  the 
contractor’s  readiness  to  begin  formal  CSCI 
testing  in  accordance  with  the  software  test 
procedures  (Hardware  doesn’t  conduct  such 
a  review,  but  should  highly  consider  doing 
so).  Systems  engineering  activities  also 
include  auditing  engineering  documentation 
(drawings  and  specifications),  auditing  system 
test  activities  (test  procedures,  set-ups,  and 
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data),  configuration  control  activities,  and 
completion  of  the  verification  process. 

The  FSD  phase  provides  verification  of 
operational  effectiveness  and  suitability 
before  deployment  by  testing  the  system  or 
equipment  in  its  intended  operational  and 
support  environment.  The  test  results  are 
evaluated  in  reviews  and  audits  intended  to 
confirm  that  the  system  design  is  sufficiently 
mature  to  proceed  with  production  and 
support  activities  that  initiate  operational  use. 

A  Functional  Configuration  Audit  (FCA) 
is  conducted  on  each  Cl  before  Milestone 
III.  The  Cl  must  represent  the  configuration 
released  for  production,  and  demonstrate 
compliance  with  the  development 
specifications  (Type  B).  Each  Cl  is  also 
subjected  to  a  Physical  Configuration  Audit 
(PCA).  The  PCA  may  be  accomplished  in 
the  FSD  phase,  but  is  usually  done  in  the 
beginning  of  the  Production  phase  on  the 
first  deliverable  Cl  that  is  built  on  production 
tooling.  Once  it  has  been  established  that 
each  production  article  is  built  in  accordance 
with  the  product  specifications  (Type  C),  the 
PCA  is  complete.  After  PCAs  for  all  the 
CIs  are  completed,  a  system  level  PCA  is 
accomplished  and  the  product  baseline  for 
the  system  is  established.  A  system  Formal 
Qualification  Review  (FQR)  using 
operational  testing  and  evaluation 
information  is  held  at  the  end  of  the  FSD 
phase  or  at  the  earliest  time  that  adequate 
test  results  become  available. 

The  output  of  FSD  is  a  tested  design  that 
meets  contract  requirements  and  the 
documentation  necessary  to  enter  the  Full 
Rate  Production/Deployment  (P/D)  and  the 
Operation  and  Support  (O/S)  phases. 


Included  are  the  product,  process,  and 
material  specifications  (Types  C,  D,  and  E); 
Production  Plan;  ILSP;  CRLCMP;  and  an 
RFP  for  the  Production  phase.  Leader  and 
follower  contractors  are  selected  and  second 
source  qualification  procedures  are 
established  as  called  for  by  the  program’s 
acquisition  strategy.  Each  program  requires 
a  DCP  update  and  SSARC  review.  A  DAB 
review  also  occurs  if  the  production  decision 
has  not  been  previously  delegated  to  the 
service  or  if  Milestone  II  cost  thresholds  are 
exceeded. 

23.5  Production/Deployment  Phase 

The  primary  objective  of  the  Production 
phase  is  to  produce  and  deliver  an  effective, 
fully  supported  system  at  an  optimal  cost.  In 
a  production  run  where  many  items  are  to 
be  delivered,  manufacturing  is  usually 
accomplished  in  two  segments.  The  first 
segment  starts  with  low-rate  production  of 
initial  product  batches  or  blocks.  During  the 
second  segment,  the  rate  increases  to  peak 
rate  production  as  necessary  changes 
resulting  from  initial  operat  onal  use, 
experience,  review,  audits,  testing,  and 
production  experience  are  incorporated. 

23.6  Operation  and  Support  (O/S)  Phase 

The  O/S  phase  starts  with  deployment  of 
the  system  and  continues  until  disposal 
(which  marks  the  end  of  the  system  life 
cycle).  The  major  activities  dur'ng  this 
period  include  introducing  modifications  and 
product  improvements  as  necessary 
throughout  deployment  as  well  as  supporting 
the  fielded  system  with  items  such  as  tools, 
spare  parts,  and  technical  documents. 
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CHAPTER  3 


SYSTEMS  ENGINEERING  MANAGEMENT  PLAN 


3.1  INTRODUCTION 

The  basic  plan  governing  the  systems 
engineering  effort  is  the  System  Engineering 
Management  Plan  (SEMP).  The  SEMP  is  a 
concise  top  level  technical  management  plan 
for  the  integration  of  all  systems  engineering 
activities.  Systems  engineering,  basically 
composed  of  two  components,  systems 
engineering  management  (SEM)  and  the 
systems  engineering  process  (SEP),  is 
implemented  through  the  SEMP.  Figure  3-1 
depicts  the  basic  relationship  between  the 
SEMP  and  the  SEM/  SEP  activities. 

The  purpose  of  the  SEMP  is  to  make 
visible  the  organization,  direction  and  control 
mechanisms,  and  personnel  for  the 
attainment  of  cost,  performance,  and 
schedule  objectives.  The  who,  what,  when. 


where,  how,  and  why  of  the  systems 
engineering  activities,  including  information 
on  relevant  interfaces  and  engineering 
specialty  areas,  must  be  clearly  delineated. 
As  such,  the  SEMP  is  supported  by  a 
number  of  engineering  specialty  plans, 
illustrated  in  Figure  3-2,  that  describe  the 
technical  activities  for  each  of  these  critical 
areas.  The  principal  role  of  the  SEMP, 
therefore,  is  use  as  a  management  tool  in 
identifying  and  assuring  the  control  of  the 
overall  systems  engineering  process. 

The  SEMP  should  be  the  primary 
document  used  in  evaluating  a  contractor’s 
technical  proposal.  As  a  minimum,  it  should 
1)  reflect  the  engineering  management 
procedures/  practices  of  the  contractor,  2) 
define  the  system/  subsystem  integration 
requirements  for  the  interfaces  and  their 
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SYSTEMS  ENGINEERING  MANAGEMENT  PLAN 
Figure  3-2 


relationships  with  the  engineering  specialties,  the  standard  format  defined  found  in 
discussed  in  Chapter  4  of  this  guide;  and  3)  MIL-STD-499A  [1],  or,  if  accepted  by  the 

reflect  tailoring  of  documentation  and  government  Program  Manager  (PM),  any 

technical  activities  to  meet  program  peculiar  contractor  proposed  format  which  provides 
requirements.  all  necessary  information.  The  standard 

format  has  three  parts:  Part  I,  'Technical 
3.2  CONTENT  OF  THE  SEMP  Program  Planning  and  Control";  Part  II, 

"Systems  Engineering  Process";  and  Part  III, 
MIL-STD-499A  [1]  was  developed  to  "Engineering  Specialty  Requirements".  Data 
assist  in  defining  the  systems  engineering  item  description  (DID),  DI-S-3618/  S152  [2], 
effort  in  support  of  DoD  acquisition  describes  the  format,  content,  and 

programs.  It  stipulated  that  a  SEMP  shall  be  preparation  instructions  for  a  SEMP  to  satisfy 

submitted  as  a  separate  and  complete  entity  MIL-STD-499A  [1]  requirements.  It  should 
within  the  contractor’s  proposal  describing  be  noted  that  this  DID  is  based  on 

how  a  fully  integrated  engineering  effort  will  MIL-STD-499  and  has  not  been  updated  to 

be  managed  and  conducted.  SEMPs  may  use  the  format  defined  in  MIL-STD-499A  [l]. 
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Some  tailoring  of  the  DID  is  necessary 
before  including  it  in  the  Contract  Data 
Requirements  List  (CDRL);  however,  DID, 
UDI-E-23974  [3]  should  be  used  when  a 
contractor  proposed  SEMP  format  is 
acceptable.  In  any  case,  a  typical  SEMP 
should  contain  the  information  listed  in  the 
sample  format  of  Figure  3-3. 

3.2.1  Technical  Program  Planning  and 
Control 

Part  I,  'Technical  Program  Planning 
and  Control",  describes  the  contractor’s 
proposed  process  for  the  planning  and 
control  of  the  engineering  efforts  for  the 
system’s  design,  development,  test,  and 
evaluation.  It  identifies:  1)  the  contractor’s 
organizational  responsibilities  and  authority 
for  SEM;  2)  the  contractor’s  control  of 
subcontracted  engineering,  verification, 
configuration  management,  and  technical 
document/  data  management;  and  3)  the 
proposed  plans  and  schedules  for  technical 
design  and  program  reviews.  The  following 
is  a  list  of  recommended  areas  to  be 
addressed  in  Part  I  of  the  SEMP: 

a.  Program  Risk  Analysis 

b.  Engineering  Program  Integration 

c.  Contract  Work  Breakdown  Structure 

d.  Assessment  of  Responsibility  and 
Authority 

e.  Program  Reviews 

f.  Technical  Design  Reviews 

g.  Technical  Performance  Measurement 


h.  Interface  Control 

i.  Documentation  Control 

j.  Plan  for  other  technical  and  program 
management  tasks. 

3.2.2  System  Engineering  Process 

Part  II,  "Systems  Engineering  Process" 
describes  the  contractor’s  proposed  systems 
engineering  process  used  in  defining  the 
system  design  and  test  requirements.  This 
part  explains  the  contractor’s  intended 
strategy  for  generating  multiple  alternative 
designs  at  each  development  level,  ,  d  the 
trade-off  results  which  trigger  iteration  of  the 
system  design  process.  It  shall  include  the  1) 
specific  tailoring  of  the  process  to  the 
requirements  of  the  system;  2)  procedures 
to  be  used  in  implementing  the  process;  3) 
trade  study  methodology;  4)  types  of 
mathematical  or  simulation  models  to  be 
used  for  system  and  cost  effectiveness 
evaluations;  5)  generation  of  specifications; 
and  6)  generation  of  applicable  engineering 
documentation.  The  following  is  a  list  of 
elements  of  the  systems  engineering  process 
to  be  addressed  in  Part  II  of  the  SEMP: 

a.  Functional  Analysis 

b.  Requirements  Allocation 

c.  Trade  Studies 

d.  Design  Optimization/  Effectiveness 
Analysis 

e.  Synthesis 

f.  Technical  Interface  Compatibility 
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lyitaa  Engineering  Managamant  Plan 

T<tl«  Page 
Introduction 

Part  1  Technical  Program  Planning  and  Control 
t.O  Responsibilities  and  Authority 

1.1  Standard*,  Procedure*,  and  Training 

1.2  Progaai  Risk  Analyai* 

1.3  Work  areekdown  Structure* 

1.4  Prograai  lav  lews 

1.5  Technical  Review* 

1.6  Technical  Performance  M«a*ur**nnt* 

1.7  Change  Control  Procedures 

1.8  Engineering  Prograa  Integration 

1.9  Interface  Control 

1.10  Milestones/  Schedule 

1.11  Other  Plane  and  Controls 

Part  2  Systems  Engineering  Process 

2.0  Mission  and  Requirements  Analysis 

2.1  Functional  Analysis 

2.2  Requirements  Allocation 

2.3  Trade  Studies 

2.4  Design  Optimization/  Effectiveness  Compatibility 

2.5  Synthesis 

2.6  Technical  Interface  Compatibility 

2.7  Logistic  Support  Analysis 

2.8  Producibility  Analysis 

2.9  Specification  Tree/  Specifications 

2.10  Documentation 

2.11  Systems  Engineering  Tool* 

Part  3  Engineering  Specialty/  Integration  Requiraamnts 

3.1  Integration  Destgn/'Plana 

3.1.1  Reliability 

3.1.2  Maintainability 

3.1.3  Huaan  Engineering 

3.1.4  Safety 

3.1.5  Standardization 

3.1.6  Survivability/  Vulnerability 

3.1.7  Electromagnetic  Coagmtibility/lnteference 

3.1.8  Elecromagnatic  Pulse  Hardening 

3.1.9  Integrated  logistics  Support 

3.1.10  Computer  Resources  Life  Cycle  Management  Plan 

3.1.11  Producibility 

3.1.12  Other  Engineering  Specialty  Requiroments/Plsns 

3.2  Integration  System  Test  Plans 

3.3  Coapatibility  with  Supporting  Activities 

3.3.1  System  Cost  Effectiveness 

3.3.2  Value  Engineering 

3.3.3  TQM/  Quality  Assurance 

3.3.4  Materials  and  Processes 


Typical  SEMP  Format 
Figure  3-3 
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g.  Logistics  Support  Analysis 

h.  Producibility  Analysis 

i.  Generation  of  Specifications 

j.  Other  Systems  Engineering  Tasks 
3.2.3  Engineering  Specialty  Integration 

Part  III,  "Engineering  Specialty 
Integration"  describes  the  contractor’s 
proposed  efforts  to  integrate  the 
requirements  of  the  engineering  specialties, 
such  as  reliability,  maintainability,  human 
engineering,  producibility,  survivability/ 
vulnerability,  electromagnetic  interference/ 
comparability  (EMI/EMC),  safety, 
electromagnetic  pulse  hardening  (EMP), 
logistics  engineering,  environmental 
considerations,  and  other  areas  into  the 
mainstream  system  design  effort.  It  will 
include  a  summary  of  each  of  the  specialty 
programs  and  cross  reference  the  individual 
plans  covering  such  specialty  programs.  This 
part  shall  depict  the  integration  of  the 
specialty  efforts  and  parameters  into  the 
systems  engineering  process  and  show  their 
consideration  during  each  iteration  of  the 
process.  When  the  specialty  programs 
overlap,  the  SEMP  shall  define  the 
responsibilities  and  authorities  of  each. 

33  IMPLEMENTATION  OF  THE  SEMP 

MIL-STD-499A  [X]  requires  that  an 
approved  SEMP  be  implemented  with  the 
award  of  the  Full  Scale  Development  (FSD) 
contract.  The  SEMP  should  be  reviewed  as 
a  part  of  the  FSD  source  selection  process, 
modified  during  negotiations,  and 
implemented  contractually  at  the  earliest 


possible  date.  However,  some  of  the  most 
critical  systems  engineering  activity  will  have 
been  completed  before  that  time  and  a  PM 
cannot  reasonably  wait  to  gain  visibility  into 

a  contractor’s  systems  engineering  methods. 
There  are  several  ways  for  the  PM  to 
improve  visibility  during  the  early  phases  of 
the  program.  Keep  in  mind,  however,  that 
(1)  contractors  have  their  own  unique  set  of 
business  standards/  procedures;  (2) 
contractors  have  their  own  version  of  a 
system  concept;  (3)  the  acquisition  strategy 
for  each  program  is  different  and  that  the 
contractors  utilized  in  the  Concept 
Exploration/  Definition  (C/E)  phase  may  not 
even  be  considered  for  the  FSD  effort;  and 
(4)  all  programs  do  not  necessarily  start  with 
the  C/E  phase,  but  may  start  with  the 
Concept  Demonstration/  Validation  (D/V)  or 
FSD  phase  of  the  acquisition  life  cycle.  In 
other  words,  each  program  has  its  own 
peculiar  set  or  requirements  and  the  PM 
must  structure  the  development,  delivery,  and 
implementation  requirements  for  a  SEMP 
accordingly. 

Generally  speaking,  the  government 
should  review  each  offeror’s  systems 
engineering  performance  on  previous 
comparable  programs  and  evaluate  the 
offeror’s  plan  for  control  and  execution  of 
C/E  phase  systems  engineering  tasks  as  a 
part  of  the  C/E  phase  source  selection 
process.  A  task  can  be  included  in  the  C/E 
phase  statement  of  work  (SOW)  to  require 
the  contractor  to  generate  an  initial  draft 
SEMP  for  the  program  as  an  end-of-phase 
deliverable.  This  plan  is  intended  to  be 
general  in  scope  and  to  contain  only  essential 
details  regarding  timing  and  approval  of 
major  technical  management  documentation. 
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Since  contractors  will  need  a  SEMP  for 
internal  use  to  execute  staffing,  organization, 
and  analysis  tasks  in  establishing  their  D/V 
effort  requirements,  the  delivery  of  such  a 
document  for  government  review  would  not 
adc  ignificantly  to  the  C/E  phase  effort. 

The  government  may  elect  to  use  the 
initial  draft  SEMPs  as  part  of  the  source 
selection  for  the  D/V  phase  contract  or  may 
require  an  updated  version  of  it  to  be 
delivered  as  part  of  the  D/V  proposal.  In 
either  case,  the  SEMP  should  reflect,  in 
detail,  how  the  contractor  plans  to  control 
the  systems  engineering  activities  during  D/V. 
When  the  D/V  phase  will  include  major  or 
potentially  permanent  design  efforts,  the 
SEMP  should  be  delivered,  reviewed,  and 
approved  by  the  government  PM,  and 
imp'<  nented  through  a  D/V  phase  contract 
modification  or  option  execution,  before  this 
design  activity  begins.  The  PM  may  also 
require  expansion  of  the  SEMP  into  a  formal 
standard  format  (DI-S-3618/S-152  [2])  or  an 
approved  contractor  format  (UDI-E-23974 
[3])  SEMP.  Recognizing  that  a  formal 
standard  format  SEMP  will  usually  be 
require-'’  1  FSD  proposals,  delivery  in  that 
formal  mg  D/V  may  result  in  savings  on 
the  tota  rmlractor  administrative  effort. 

As  stated  previously,  the  SEMP  is 
required  tc  je  approved  prior  to  or  in 
conjunction  vith  the  commencement  of  FSD 
activity.  Ft  competitive  acquisition,  each 
contractor  will  develop  and  implement  their 
own  unique  SEMP.  The  SEMP,  which  may 
be  used  during  any  phase,  should  include 
any  update  F.ans  specifying  which  sections 
must  be  kept  current,  who  will  submit  and 
review  changes,  and  on  what  dates  or  in 
association  with  what  program  milestones  the 


updated  editions  will  become  effective. 

3.4  RELATED  PLANNING  EFFORTS 

Additional  plans  for  the  engineering 
specialties  should  be  referenced  in  Part  III  of 
the  SEMP.  The  need  for  separate 
engineering  specialty  plans  is  a  key  part  of 
the  acquisition  streamlining  effort.  Where 
separate  engineering  specialty  management 
plans  are  not  prepared,  often  much  of  the 
basic  information  and  work  will  be  described 
in  the  overall  SEMP.  Some  of  the 
engineering  specialty  plans  which  are 
normally  integrated  under  the  systems 
engineering  umbrella  are  listed  below: 

a.  Technical  performance  measurement 

b.  Producibility 

c.  Maintainability 

d.  Quality 

e.  Human  engineering 

f.  Safety 

g.  Logistic  support  analysis 

h.  Reliability 

i.  Production  engineering 

j.  Contamination  and  corrosion  control 

k.  Parts,  materials,  and  processes  control 

l.  Electromagnetic  control 

m.  Nuclear  hardening 
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n.  Vulnerability/  survivability 

o.  Weight  control 

p.  Mass  properties  control 

q.  Packaging,  handling,  storage,  and 
transportation. 

Where  engineering  specialty  plans  are 
required,  they  must  clearly  indicate  how  the 
specialty  contributes  to  and  benefits  from  the 
general  systems  engineering  process  and 
documentation.  These  detailed  plans  also 
provide  the  initial  basis  for  the  development 
and  review  of  program  cost  estimates  and 
schedules.  Although  each  specialty  plan  will 
probably  have  its  own  DID,  all  plans  should, 
at  a  minimum,  contain  the  following  systems 
engineering  interface  information: 

a.  Objective  -  Purpose  of  including  the 
specialty  and  the  scope  of  its  role  within  the 
systems  engineering  process 

b.  Activity  Definition  -  Summary  description 
of  all  tasks  required  to  fulfill  the  specified 

3.5  REFERENCES 

1.  MIL-STD-499A,  "Engineering 
Management" 

2.  DI-S-36 18/S- 152,  "System  Engineering 
Management  Plan  (SEMP)". 


function,  the  content  of  required  inputs  from 
systems  engineering,  and  the  expected 
products  to  be  provided  to  systems 
engineering 

c.  Responsibilities  -  Definition  of  all 
organizations  supporting  (or  supported  by) 
the  activity,  which  tasks  they  are  responsible 
for,  and  their  lines  of  authority,  with 
particular  emphasis  on  the  division  of 
analytical  tasks  between  the  systems 
engineering  organization  and  the  specialty 
organization 

d.  Schedules  -  Timing  and  sequence  of  all 
engineering  tasks  related  to  major  milestones 
for  system  development  and  design,  and  to 
specific  inputs  from  supporting  organizations 
in  the  systems  engineering  process 

e.  Resource  Definition  -  Identification  of 
specific  hardware,  software,  personnel,  and 
facilities  required  to  complete  the  engineering 
tasks  according  to  the  schedule  and  to 
provide  the  required  support,  from  this 
discipline’s  point  of  view,  to  the  overall 
systems  engineering  process. 


3.  UDI-E-23974,  "Plan,  Systems  Engineering 
Management  (SEMP)". 
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CHAPTER  4 


SYSTEM  INTEGRATION 


4.1  INTRODUCTION 

A  Primary  role  of  system  engineering  is 
to  ensure  that  the  many  diverse  elements 
comprising  a  system  are  compatible  and 
ready  when  needed.  This  avoids  the  situation 
in  which  hardware  or  software,  when 
integrated  into  the  system,  fails  to  function 
as  intended  as  part  of  the  system. 
Integration  ensures  that  all  the  "pieces"  of 
the  system  will  work  together  to  realize 
system  goals.  Proper  planning  and 
coordination  throughout  the  development 
process  can  ensure  that  problems  are  held 
to  a  minimum  and  that  the  final 
implementation  of  the  system  satisfies  the 
mission  performance  requirements. 

4.2  APPROACH 

The  basic  plan  for  managing  the  System 
Engineering  effort  is  the  System  Engineering 
Management  Plan  (SEMP),  which  is  defined 
in  MIL-STD-499A  [I],  and  is  prepared  in 
three  parts.  Part  One,  'Technical  Program 
Planning  and  Control",  identifies 
organizational  responsibilities  and  authority 
for  system  engineering  management,  including 
control  of  subcontracted  engineering, 
verification,  configuration  management, 
document  management,  and  plans  and 
schedules  for  design  and  technical  program 
reviews.  Part  Two,  "Systems  Engineering 
Process",  describes  the  process  used  in 
defining  and  allocating  requirements  and 


their  documentation.  Part  Three, 
"Engineering  Specialty  Integration",  defines 
how  the  enpineering  specialties  of 
reliability,  ma  .ntainability,  human 
engineering,  safety,  logistics,  and  other 
areas  are  integ  ated  into  the  mainstream 
design  effort. 

The  contractor  begins  preparing  the 
SEMP  in  the  Concept  Exploration/ 
Definition  (C/E)  phase,  defining  how  he 
will  structure  his  organization  for  the 
specific  program  and  how  he  will  control 
the  total  engineering  process  to  provide 
a  product  that  satisfies  performance 
requirements.  The  SEMP  will  define  the 
models  and  simulations  that  are  used  for 
defining  system  requirements  and 
optimizing  system  configuration.  These 
same  models  are  employed  in  the 
verification  process,  which  is  also  in  the 
SEMP.  Appendix  A  to  MIL-STD-499A 
[1]  defines  specific  tasks  to  be 
accomplished  under  Parts  One  and  Two. 
These  tasks  may  be  tailored  depending 
on  the  nature  of  the  program. 

The  SEMP  should  be  program  specific 
and  identify  the  organizational 
configuration,  functions  and 
responsibilities,  management  techniques, 
analyses,  trade  studies,  simulations, 
Technical  Performance  Measurements 
(TPM)  parameters,  and  schedules,  that 
will  be  investigated  or  employed  on  the 
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program.  During  negotiations  and  after 
contract  award,  the  SEMP  may  be  revised 
to  incorporate  or  delete  items  the 
government  and  contractor  agree  are 
suitable.  Thereafter,  the  SEMP  provides  the 
basis  for  all  contractor  system  engineering 
effort  on  the  program. 

During  the  Concept  Demonstration/ 
Validation  (D/V)  phase,  plans  for  critical 
specialty  areas  are  also  prepared.  These 
plans  are  summarized  or  referenced  in  Part 
Three  of  the  SEMP.  Where  no  separate 
plans  are  prepared,  a  summary  of  the  effort 
is  included  in  the  SEMP.  All  plans  should 
contain  the  following  dements: 

a.  Objective:  Purpose  and  scope. 

b.  Activity  Defmition:  Summary  description 
of  all  tasks  required  to  fulfill  the  specified 
function  including  inputs  and  nature  of 
outputs. 

c.  Responsibilities:  Definition  of  all 
organizations  supporting  activity,  portion  of 
activity  for  which  they  are  responsible,  and 
line  of  authority. 

d.  Schedule  of  Activities:  Time  sequence 
of  tasks  tied  to  program  schedule  milestones, 
showing  inputs  from  supporting 
organizations. 

e.  Resource  Definition:  Identification  of 
specific  hardware,  software,  and  facilities 
required  to  perform  this  activity  within  the 
specified  time  frame. 

The  level  of  detail  that  can  be  provided 
in  the  plans  reduces,  to  a  significant  extent, 
the  number  of  problems  that  will  be 
encountered  in  performing  this  activity.  In 


addition,  the  detail  provides  a  firm  basis 
for  the  development  of  program  cost  and 
schedules.  These  plans  may  include: 

a.  Reliability  Plan 

b.  Maintainability  Plan 

c.  Human  Engineering  Plan 

d.  Safety  Plan 

e.  Contamination  and  Corrosion  Control 
Plan 

f.  Parts,  Materials,  and  Processes 
Control  Plan 

g.  Electromagnetic  Control  Plan 

h.  Survivability/  Vulnerability  Plan 

43  ENGINEERING  SPECIALTY 
INTEGRATION 

Engineering  specialties  are  those 
disciplines  which  support  the  design 
process  by  applying  knowledge  from  a 
specific  area  to  ensure  system  operability 
in  its  operational  environment.  They 
include  reliability,  maintainability,  human 
engineering,  safety,  electromagnetic 
compatibility,  parts/  materials/  processes 
and  other  specialist  areas  involved  in 
development  of  a  general  class  (ships, 
aircraft,  tanks)  of  system.  These 
specialties  are  integrated  into  the 
development  effort  through  the  system 
engineering  process.  MIL-STD-499A  [1] 
speaks  to  this  effort  as  follows: 

"System  Engineering  indudes  the 
integration  of  the  engineering  specialties  and 
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the  management  of  a  totally  integrated  effort 
of  design  engineering,  specialty  engineering  test 
engineering,  and  production  engineering  to 
ensure  their  influence  on  design ." 

In  Section  3.4  of  MIL-STD-499A  [1], 
engineering  specialty  integration  is  defined 
as  follows: 

' The  timely  and  appropriate  intermeshing  of 
engineering  efforts  and  disciplines  such  as: 
reliability,  maintainability,  logistics  engineering, 
human  factors,  safety,  value  engineering 
standardization,  transportability,  etc,  to  ensure 
their  influence  on  design." 

43.1  Integration  Framework 

A  conceptual  framework  for  the 
discussion  of  engineering  integration  is  shown 
in  Figure  4-1.  It  shows  a  way  to  categorize 
the  disciplines  that  participate  in  the  systems 
engineering  process.  Although  "traditional" 
and  "specialty"  engineering  are  shown  in 
separate  branches,  both  branches  are  equally 
important  parts  of  conceptual,  prcliminaiy, 


and  detailed  design.  The  categorization 
simply  reflects  that,  because  the  design 
could  be  accomplished  without  considering 
the  "specialty"  disciplines,  special  emphasis 
must  be  placed  on  the  detailed  integration 
of  these  efforts.  All  branches  shown  are 
representative  but  not  complete. 

The  systems  engineering  process 
provides  a  technical  management 
framework  for  the  design  team.  At  any 
point  in  the  acquisition  phase  the  design 
team  consists  of  a  mix  of  traditional 
engineers,  "specialty"  engineers,  test 
engineers,  logistics  engineers,  value 
engineers,  production  engineers,  etc.  In 
the  earlier  stages  of  the  program,  the  team 
may  be  engineers  skilled  in  functional 
analysis.  During  full  scale  deveolpment 
(FSD),  the  composition  of  the  team  will 
shift  to  traditional  engineers  skilled  in 
traditional  engineering  design  practices  and 
specialty  engineers  skilled  in  design 
support.  During  late  FSD  and  production 
phases,  the  emphasis  will  shift  to  a  team 
dominated  by  production,  logistics,  and  test 


Figure  4-1 

Engineering  Integration  in  the 
Systems  Engineering  Framework 


engineers. 

The  integration  of  the  design  process  is 
conceptualized  in  Figure  4-2,  Three 
integration  filters  are  shown  in  the  overall 
process.  As  information  is  taken  into  the 
"traditional  discipline  filter",  emphasis  is 
placed  on  those  traditional  design  techniques 
(such  as  stress  analysis  of  structures)  required 
at  a  given  stage  of  design  (conceptual, 
preliminary,  or  detailed  design).  The 
traditional  design  engineers  draw  heavily  on 
the  state  of  the  art  technology  of  their  area. 
At  the  same  time,  design  documentation  is 
being  developed  and/  or  modified  by 
"specialists"  in  such  areas  as  reliability, 
maintainability,  and  human  engineering.  The 
"specialists"  are  establishing  requirements 
independent  of  the  emerging  traditional 
discipline  design,  but  they  are  also  reviewing 
and  modifying  the  traditional  discipline 
output.  The  SEMP  would  normally  contain 


enough  detail  to  show  the  timing  and 
format  of  data  which  each  "specialist" 
would  supply  to  the  systems  engineering 
process.  Finally,  all  requirements  are 
filtered  by  the  unique  demands  of  the 
products  which  comprise  the  system.  The 
requirements  are  then  described  by 
specifications  and  drawings  which  set  out 
in  clear  language  the  "design-for" 
requirement.  Of  course,  the  diagram  in 
Figure  4-2  is  only  conceptual,  as  the 
design  of  any  product  element  usually  will 
require  several  iterations  through  each 
process  step. 

Specialty  engineers  draw  upon  an 
extensive  background  of  data  extracted 
from  past  and  current  programs  to 
develop  standards,  guidelines,  and 
checklists  to  support  and  evaluate 
development  of  new  designs.  To  maintain 
proficiency  in  their  specialty  area  and  to 


Figure  4-2 

Technical  Integration  Procesi 
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ensure  rapid  communications  on  new 
developments  and  problems,  they  are 
permanently  a  part  of  the  specialty  staff  and 
temporarily  assigned  to  programs  as  the  need 
arises.  Depending  on  the  nature  of  the 
contractor’s  organizational  structure,  these 
may  be  separate  organizations  or  they  may 
be  grouped  under  major  division.  For 
example,  during  the  early  phases  of  the 
program  up  through  Preliminary  Design 
Review  (PDR)  the  specialties  may  report  to 
Systems  Engineering.  Depending  on  their 
function  and  size  within  the  program,  they 
may  then  become  separate  organizations, 
such  as  Integrated  Logistics  Support  (ILS). 

The  more  common  approach  for  large 
system  oriented  companies  is  the  matrix 
organization  in  which  personnel  are  assigned 
to  programs  from  their  "home"  organizations 
for  the  period  they  are  needed  on  the 
program.  Within  the  program,  all 
engineering  may  be  under  the  responsibility 
of  a  chief  engineer.  In  this  case  the  system 
engineering  manager  is  another  of  the 
functional  managers  (such  as  mechanical, 
electrical,  software,  etc.)  reporting  to  the 
chief  engineer.  System  engineering  is 
responsible  for  definition  of  requirements 
and  interfaces  and  integration  of  the 
hardware  and  software  into  the  system.  The 
engineering  specialties  may  be  grouped  in  a 
single  organization  and  report  to  the  chief 
engineer,  or  they  may  be  located  in  the 
system  engineering  organization,  depending 
on  the  size  and  nature  of  the  program.  In 
either  case,  the  specialists  work  with  system 
engineering  to  define  and  document 
requirements  and  work  with  the  functional 
engineering  groups  to  ensure  the  necessary 
specialty  features  are  incorporated  in  the 
design.  The  chief  engineer  is  more  typical 
of  established  aircraft  firms.  A  different 


management  approach  which  has  arisen 
over  the  past  decade  is  that  of  the  Chief 
Systems  Engineer  (CSE).  The  CSE  is  one 
of  the  functional  managers  reporting  to 
the  program  manager  (PM).  The  primary 
role  of  the  CSE  is  that  of  defining  the 
requirements  and  auditing  the  design.  He 
is  the  PM’s  top  technical  authority  and  the 
head  of  Systems  Engineering.  Since  the 
engineering  specialties  perform  a  similar 
function,  they  are  usually  placed  within  the 
systems  engineering  organization  to 
strengthen  their  voice  in  requirements 
definition  and  documentation  and  to 
ensure  a  total  and  complete  review  of  the 
evolving  designs.  The  role  of  the  specialist 
under  system  engineering  is  to  define 
requirements  for  design  and  verification, 
to  audit  the  resulting  design  for 
compliance,  and  to  plan  all  activities 
related  to  their  functions.  Placement  of 
specialty  engineering  within  system 
engineering  assures  that  all  requirements 
on  the  system  are  identified  and  uniformly 
levied  and  flowed  down  throughout  the 
system.  In  addition,  it  assures  the 
coordination  of  system  audit  activities  and 
their  uniform  application. 

Engineering  specialists  are  brought 
into  the  design  process  at  a  very  early 
stage-in  limited  numbers  during  the  C/E 
phase  if  candidate  designs  have  been 
developed  in  sufficient  depth  to  be 
analyzed  and  in  full  complement  for  the 
D/V  phase.  Both  the  reliability  and 
maintainability  engineers  are  typically 
among  the  earliest  involved,  since  they 
can  have  the  greatest  impact  on  design. 
The  specialists  define  requirements  in  their 
area,  typically  by  tailoring  government 
standards  to  the  specific  needs  of  the 
program.  These  requirements  are  then 
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placed  in  specifications.  The  requirements 
are  both  quantitative  (e.g.,  reliability  values, 
and  allocations,  Mean-Time-to-Repair  values, 
and  availability  values)  and  qualitative  (by 
referencing  standards,  properly  tailored  in  the 
specification  appendices,  which  specify 
constraints,  procedures,  limitations,  etc.). 
Since  the  top  specifications  (system  and 
segment)  are  prepared  by  systems 
engineering  and  lower  level  ones  require 
their  review  and  approval,  this  ensures  that 
specialty  requirements  are  incorporated  at  all 
specification  levels.  Specialists  also  work  with 
program  engineering  personnel,  reviewing 
and  analyzing  the  evolving  design  and 
ensuring  the  incorporation  of  necessary 
features  in  their  area  (e.g.  redundancy,  access 
areas,  Built-in-Test  (BIT)). 

During  the  D/V  phase,  each  specialty 
area  prepares  plans  defining  how  they  will 
perform  tasks  in  the  FSD  phase,  describing 
procedures,  resources,  tasks,  and  the  schedule 
of  activities.  Although  often  identified  as 
separate  Contractor  Data  Requirements  List 
(CDRL)  items,  these  also  are  also 
summarized  in  Part  III,  Integration  of  the 
Engineering  Specialties,  of  the  SEMP.  The 
SEMP  also  discusses  how  they  are  integrated 
and  relates  the  specialty  tasks  to  the  overall 
systems  engineering  effort. 

In  all  phases,  specialists  form  a  part  of 
the  systems  engineering  design  review  team. 
Data  packages  produced  by  the  program 
engineering  organizations  are  reviewed 
against  established  check  lists  to  verify 
compliance  to  all  specialty  area  requirements. 
Deficiencies  are  documented  in  action  items 
as  part  of  the  meeting  minutes  and  followed 
up  by  systems  engineering  to  assure 
resolution.  All  change  packages  are  also 
reviewed  by  specialty  engineers  prior  to  their 


presentation  to  the  Configuration  Control 
Board  (CCB)  to  ensure  that  they  will  have 
no  deleterious  effect  on  the  system 
operability. 

Specialty  area  test  requirements  are 
also  developed.  These  may  include 
specific  specialty  tests  such  as  reliability 
life  tests,  EMC/EMI  tests,  and 
maintainability  teardown  tests,  as  well  as 
the  incorporation  of  specialty  area  tests 
into  the  general  component  (software  and 
hardware)  subsystem,  and  system  level  test 
plans.  These  form  the  Test  and 
Evaluation  Master  Plan  (TEMP)  which  is 
the  responsibility  of  systems  engineering. 
During  development  and  qualification 
testing  in  the  FSD  phase,  test  data  are 
evaluated  by  specialty  groups  to  determine 
if  required  standards  and  levels  in  their 
areas  have  been  met.  Qualification  (at  the 
component  level)  is  certified  by  reliability 
engineers  based  on  demonstrated 
capability  to  meet  specification 
requirements. 

The  specialty  tasks  are  performed 
under  the  cognizance  of  the  systems 
engineering  manager  (or  CSE)  who  directs 
the  requirements  derivation  and  flow  down 
effort,  reviews,  audits,  and  system 
verification  effort.  As  with  the 
performance  requirements  stated  in  the 
system  specification,  specialty  engineering 
requirements  must  also  be  reviewed  to 
ensure  that  they  do  not  impose  an 
unnecessary  burden  on  the  system,  but  are 
adequate  to  allow  the  system  to  meet 
operability  requirements.  It  must  be 
recognized  that  while  the  requirements 
setting  and  review  and  audit  functions  are 
typical  systems  engineering  activities, 
specialists  are  also  far  more  involved  in 
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the  detail  design  and  testing  than  is  normal 
system  engineering  practice.  This  imposes  a 
greater  burden  on  system  engineering 
management  to  assure  that  the  specialists  do 
not  "overdrive"  the  design,  i.e.,  incorporate 
more  features  in  the  design  than  are  cost 
effective.  Typically,  the  lead  engineer  or 
supervisor  of  the  design  support  group 
reports  directly  to  the  systems  engineering 
manager  (or  CSE).  The  supervisor  is 
responsible  for  integrating  the  activities  of  his 
group  and  reporting  its  activities  and 
significant  decisions  to  the  manager.  He  is 
also  responsible  for  maintaining  open 
communications  channels  with  the  "home" 
system  effectiveness  organizations  to  ensure 
that  specialists  are  alerted  to  new  problems 
and  techniques  as  they  become  known  and 
that  expert  specialty  area  consultation  is 
available  when  needed.  The  manager  must 
ensure  that  specialty  requirements  are 
properly  reviewed  by  the  systems  engineering 
staff  with  regard  to  their  impact  on  cost, 
schedule,  performance,  and  operability,  and 
then  incorporated  into  the  appropriate 
documentation.  Decisions  made  by  the 
specialty  groups  have  a  major  impact  on 
operational  costs,  which  usually  constitute 
the  largest  portion  of  system  life  cycle  costs 
(LCC).  Placement  of  them  within  system 
engineering,  during  at  least  the  initial 
program  phases  (when  the  majority  of  LCC 
are  being  committed),  helps  ensure  that  an 
optimized,  cost-effective  system  will  result. 

4.3.2  Reliability  Engineering 

Reliability  engineers  address  the  issue  of 
equipment/  system  performance  by  applying 
analytical  methods  and  historical  statistical 
data.  They  derive  a  functional  model  of 
system  performance  in  accordance  with  the 
design  and  a  mathematical  model  whose 


outputs  are  inherent  failure  distributions 
and  failure  rates.  By  analyzing  the  design 
and  applying  historical  data,  an  estimate 
of  the  probability  of  successful 
performance  (or  failure)  is  calculated  for 
the  system  and  each  segment,  subsystem, 
assembly,  and  such.  Their  analysis 
identifies  the  strengths  and  weaknesses  of 
the  design  so  that  improvements  are  made 
to  the  best  advantage.  Therefore,  a 
dynamic  communication  occurs  between 
design  and  specialty  engineers  during 
concept  studies,  trade  off  analyses,  design, 
and  development. 

One  may  question  the  accuracy  of  such 
estimates,  or  in  statistical  terms,  the 
probability  density  function.  How  well 
does  the  historical  data  that  were  used 
apply  to  this  system,  this  design,  this 
mission,  and  also  how  many  trials, 
attempts,  or  missions  are  represented  by 
the  data?  Both  qualitative  and  statistical 
answers  might  be  available,  but  is 
advisable  to  examine  the  planned  testing 
program  to  determine  if  applicable, 
realistic  data  will  be  available  to  apply  to 
this  question,  and  if  the  test  program 
could  be  improved 

Reliability  estimates  based  on  the 
inherent  (generic)  failure  rates  are  useful 
for  planning  purposes,  for  comparing 
alternatives,  and  for  assessing  proposed 
changes.  Later,  when  test  and  operational 
data  are  available,  they  will  become  the 
basis  for  program  decisions  and  actions 
and  a  basis  for  revised  reliability  estimates. 

Reliability  engineering  tasks  include 
the  following: 

a.  Monitor/  Control  of  sub-contractors 
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and  Suppliers 

b.  Program  Reviews 

c.  Failure  Reporting,  Analysis  and 
Corrective  Action  System  (FRACAS) 

d.  Failure  Review  Board 

e.  Reliability  Modeling 

f.  Reliability  Predictions 

g.  Failure  Modes,  Effects  and  Criticality 
Analysis  (FMECA) 

h.  Sneak  Circuit  Analysis 

i.  Electronic  Parts/  Circuits  Tolerance 
Analysis 

j.  Reliability  Critical  Items 

k.  Effects  of  Functional  Testing,  Storage, 
Handling,  Packaging,  Transportation  and 
maintenance 

l.  Environmental  Stress  Screening 

m.  Reliability  Development/Growth  Test 
(RDGT)  Program 

n.  Reliability  Qualification  Test  Program 

o.  Production  Reliability  Acceptance  Test 
(PRAT). 

Descriptions  and  discussions  of  these 
reliability  tasks  can  be  found  in  the  Task 
Section  and  Appendix  A  to  MIL-STD-785 
[2]. 

The  reliability  requirements  of  space  and 


missile  systems  may  differ  significantly 
from  ground,  flight,  or  shipboard  systems. 
Space  and  missile  systems  are  generally 
maintainable  in  a  readiness  mode,  but  not 
maintainable  in  a  mission  mode. 
Therefore,  the  emphasis  given  to  certain 
reliability  tasks  can  vary. 

The  reliability  program  plan  is 
normally  submitted  as  part  of  the  bidder’s 
response  to  the  request  for  proposal. 
Unless  it  is  necessary  to  obtain  a  detailed 
plan  to  evaluate  the  response,  a  brief 
description  or  preliminary  plan  may  be 
sufficient.  However,  a  program  integrated 
reliability  task  schedule  and  a  manhour 
estimate  for  each  task  are  necessary. 

433  Maintainability  Engineering 

Maintainability  engineers  address  the 
maintenance  concept/  policy  as  it  is 
reflected  in  design  provisions  for  fault 
prevention,  detection,  isolation  and 
correction,  and  the  implementation 
requirements  in  terms  of  skills,  test 
equipment,  time  to  repair/  replace/  restore, 
and  maintenance  cost  over  the  life  cycle 
of  the  product.  Maintenance  concepts  are 
based  on  operability  considerations  and  on 
operations  phase  support  concepts. 
Maintenance  provisions  are  an  important 
design  factor  in  determining  system 
availability  and  in  life  cycle  cost. 
Maintainability  specialists  translate  broad 
support  concepts  and  requirements  into 
detailed  concepts  and  plans  for  each  item 
at  each  level  of  maintenance:  organization, 
intermediate,  and  depot.  A  system 
maintenance  model  is  used  to  examine 
alternative  configurations,  methods,  and 
test  techniques  to  minimize  downtime  and 
maintenance  cost  and  to  allocate 
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requirements  to  maintainable  items.  From 
these  analyses  both  qualitative  and 
quantitative  design  criteria  are  established 
so  that  design  engineers  will  incorporate 
maintainability  provisions  together  with 
configuration  and  performance  in  the  earliest 
stages  of  design.  As  design  details  emerge, 
reliability  specialists  provide  estimates  of 
maintenance  frequency-  mean  time  between 
maintenance  (MTBM)  for  the  maintainable 
items,  and  maintainability  specialists 
determine  the  restoration  times  (MTTR),  the 
direct  manhours  per  maintenance  action 
(DMH/MA),  and  the  maintenance  manpower 
cost  for  each  item  and  for  the  system.  These 
are  compared  to  the  item  allocations  and 
system  requirements.  Where  discrepancies 
exist,  they  may  require  reallocation  of 
requirements  or  a  design  modification  to 
meet  specification.  Results  are  reported  to 
reliability  engineers  and  other  specialists  as 
well  as  personnel  involved  in  life  cycle  cost 
(LCC),  design  to  cost  (DTC)  and  integrated 
logistic  support  (ILS).  Similar  tasks  and 
results  are  necessary  for  subcontract  and 
government  furnished  items.  Maintainability 
engineers  must  actively  interface  with  human 
engineering,  test  equipment  design 
(testability,  skills,  fault  isolation),  system 
safety  (hazards,  critical  items),  manufacturing 
(repair  time),  and  others  to  realize  an 
effective,  optimized  system  design. 

System  failure  diagnosis,  fault  isolation, 
removal,  replacement/repair  and  retest  or 
verification  times  are  difficult  to  estimate. 
Therefore,  maintainability  demonstration  tests 
may  be  necessary  to  obtain  useful  estimates. 
Maintainability  specialists  plan  and  direct 
these  tests  as  required  in  accordance  with 
MIL-STD-470  [3]. 

The  maintainability  program  plan  is 


normally  submitted  as  pent  of  the  bidders’ 
response  to  the  procuring  activity’s  request 
for  proposals.  In.  many  instances,  a  brief 
preliminary  plan  is  sufficient  for  proposal 
purposes,  provided  a  program  integrated 
maintainability  task  schedule  and  a 
manhour  estimate  for  each  task  are 
submitted. 

43.4  System  Safety 

Safety  specialists  analyze  the  system/ 
program  for  hazards  to  personnel  and 
equipment  and  take  action  to  eliminate  or 
control  them.  Safety  concerns  encompass 
all  personnel  and  equipment  which  may  be 
affected  by  program  plans  and  operations. 
These  include,  but  are  not  limited  to 
manufacturing,  testing,  packaging, 
handling,  transportation,  storage,  and 
government  personnel  and  equipment  at 
launch,  test,  and  operational  sites.  Local, 
state,  and  federal  laws  and  regulations  that 
are  designed  to  protect  employees  and  the 
general  public  alike  are  involved,  as  well 
as  specific  safety  requirements  of  the 
equipment  or  the  design. 

Safety  specialists  apply  formal  methods 
of  hazard  identification  and  analysis, 
develop  design  criteria,  review  the  design 
for  compliance,  and  provide  safety 
certification  of  the  equipment/  system  for 
the  readiness  review.  Safety  requirements 
and  standards  are  applied  to  all  operations 
by  performing  an  Operational  Hazard 
Analysis  (OHA)  and  reviewing  plans  and 
procedures.  Safety  critical  items  and 
operations  are  identified  and  controlled  to 
reduce  hazards  to  an  acceptable  level  of 
risk.  During  safety  critical  operations, 
safety  specialists  are  on-site  to  assure  that 
safe  procedures  and  methods  are  used  and 
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to  assist  operations  personnel  as  necessary. 

When  quantitative  safety  requirements 
are  used,  fault  tree  analysis  is  employed  to 
establish  combinations  of  modes  or  conditions 
that  produce  hazards,  combinations  of 
contributory  events,  and  the  probability  that 
the  undesired  event  will  occur.  Fault  tree 
analysis  shows  in  tree  form  the  paths  which 
could  lead  to  a  hazardous  condition  as  these 
various  occur.  A  probability  is  usually 
assigned  at  each  junction.  Lessons  learned 
from  previous  developments  are  of 
paramount  importance  to  safety.  Extensive 
records  of  accident  reports,  safety 
publications,  analysis  reports,  failure  and 
corrective  action  reports,  safety  surveys, 
audits,  and  design  reviews  are  required.  An 
Accident  Risk  Assessment  Report  (ARAR) 
may  be  prepared  to  identify  design  and 
operating  limits  to  be  imposed  on  system 
elements  to  preclude  or  minimize  mishaps 
which  could  cause  injury  or  death.  System 
safety  program  requirements  are  selected 
from  MILrSTD-882.  A  detailed  list  of  system 
safety  program  requirements  for  each  of  the 
life-cycle  phases  is  provided  in  MIL-STD-882 
[4],  paras  4.2.1  through  4.2.4. 

43.5  Parts  Engineering 

Parts  are  the  building  blocks  of 
equipment  and  systems.  The  inherent  or 
generic  failure  rates  of  parts  and  the  design 
application  of  parts  will  principally  determine 
failure  frequency,  readiness,  mission  success, 
maintenance  costs  and  logistic  support  costs. 
Parts  which  are  similar  functionally  are 
available  with  a  wide  range  of  inherent 
failure  rates  (e.g.,  10-2  failures  per  1000  hour 
to  1(M*  failures  per  1000  hour).  Depending 
upon  the  parts  selected,  a  system  of  1000 
parts  may  have  a  mean  time  between  failure 


(MTBF)  of  100  hour  or  100,000  hour. 
Established,  reliable  parts  with  the  best 
available  failure  rates  cost  more  to 
manufacture  and  test  than  commercial 
parts,  so  the  initial  cost  is  higher,  but 
system  life  cycle  cost  may  be  significantly 
lower.  Parts  are  derated  (applied  to  the 
design  at  levels  well  below  maximum 
ratings;  e.g.,  voltage,  current,  and 
temperature)  to  reduce  the  risk  failure. 

Early  in  program,  parts  specialists 
establish  the  Program  Parts  Selection  List 
(PPSL),  which  designers  use  to  select 
standard  parts  that  meet  program 
qualification,  documentation,  and  reliability 
requirements.  If  the  design  requires  parts 
not  previously  qualified,  documented,  or 
without  adequate  performance  history, 
then  these  additional  tasks  and  tests  must 
be  planned  and  completed.  Coordinated 
standards  and  coordinated  procurement  of 
parts  between  contractors  and 
subcontractors  may  be  required  to  reduce 
purchase  costs  or  improve  lead  time. 

A  Parts  Control  Board  (PCB)  may  be 
required  to  control  the  parts  program. 
The  PCB  is  composed  of  parts  control 
engineers  and  reliability  engineers 
reporting  to  systems  engineering,  program 
design  engineers,  and  product  assurance 
engineers.  They  establish  the  PPSL, 
review  and  approve  proposed  additions, 
and  define  parts  testing  requirements  and 
qualification  criteria.  The  PCB  also 
defines  criteria  for  subcontractors  and  in- 
house  testing  of  parts,  documentation  of 
parts  manufacture,  and  in-process 
monitoring  and  reporting  of  parts  assembly 
lines.  The  latter  may  include  certification 
of  operators  and  inspectors,  destructive 
physical  analysis  on  selected  samples. 
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protection  of  sensitive  surfaces,  inspection 
requirements,  and  traceability 
implementation. 

Parts  control  program  requirements  are 
selected  from  MIL-STD-965  [5].  In  many 
instances  it  may  be  necessary  to  complete 
certain  trade  studies  on  performance, 
reliability,  logistics,  and  life-cycle  cost  before 
the  parts  requirements  can  be  determined. 
The  parts  program  for  the  C/E  and  D/V 
phases  may  be  the  basis  for  developing  a 
more  specific  parts  program  for  the  FSD, 
Production  and  Deployment/Operations 
phases. 

4-3.6  Human  Engineering 

Human  engineering  specialists  address 
the  people-equipment  interface.  They  apply 
principles  of  human  capability  to  reach,  lift, 
see,  communicate,  comprehend,  and  act  to 
the  functions  and  circumstances  required. 
They  are  another  team  member  in  the  design 
process  whose  goal  is  to  optimize  the  system. 
They  first  allocate  system  functions  to 
personnel,  equipment,  software,  or  facilities. 
The  level  of  involvement  and  criticality  of 
personnel  tasks  are  identified.  Human 
factors  engineers  then  use  task  analysis  and 
time  line  studies  to  determine  if  human 
capabilities  will  be  exceeded.  They  prepare 
models  and  mock-ups  to  evaluate  alternative 
designs  or  concepts  and  for  dynamic 
simulation  of  critical  human  performance. 
Human  engineering  specialists  work  with 
design,  system  safety,  maintainability,  testing, 
training  production,  subcontractors, 
deployment,  logistic  support,  and  operations 
personnel.  The  protection  of  personnel  from 
hazardous  environmental  conditions  is  an 
important  consideration  on  which  they  help 
focus  program  attention.  Human  engineering 


program  requirements  are  usually  selected 
from  MIL-STD-46855  [6]  or  MIL-STD- 
1472  [7J. 

43.7  Electromagnetic  Compatibility  and 
Electromagnetic  Interference 

Unintentional  electromagnetic  radiation 
can  cause  unacceptable  degradation 
(interference  to  electronic  equipment 
within  range  or  initiation  of 
electroexplosive  devices).  EMC  is 
achieved  by  eliminating  or  controlling 
unintentional  radiation  to  an  acceptable 
level  or  by  shielding  equipment  from  its 
effects.  Protection  from  lightning  and 
static  charges  are  included  in  this 
discipline. 

EMC/EMI  specialists  address 
electromagnetic  sources  of  radiation  within 
the  system  such  as:  motors,  generators, 
power  sources,  signal  and  power  wiring, 
transformers,  relays,  etc.  They  develop 
design  criteria  to  minimize  potential 
radiation  shielding,  bonding,  lead  lengths, 
wire  routing,  component  placement,  and 
de-coupling.  It  is  usually  necessary  to 
perform  a  detailed  analysis  of  the 
electrical  power  system  to  determine 
power  bus  characteristics  and  dynamic 
impedance  and  to  evaluate  any  undesirable 
steady  state  or  transient  effects. 
Development  and  qualification  tests 
include  EMC/EMI  tests  to  measure 
unintended  radiation  and  its  effects. 

Requirements  for  EMI  characteristics 
of  equipment  are  set  forth  in  MIL-STD* 
461  [8J.  An  EMC  control  plan  provi4'- 
policies,  guidelines,  methods,  and  tasks 
required  to  achieve  the  characteristics 
required.  The  plan  is  usually  prepared  in 
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accordance  with  MIL-E-6051  [9]  for  systems. 


43.8  Contamination  and  Corrosion  Control 

Several  types  of  manufact-uring  require 
contamination  control.  Semi-conductors, 
microelectronics,  precision  bearing,  and  such 
require  atmospheric  control  of  airborne 
particles  and  control  of  surface  contaminants. 
Equipment  applications  and  environments 
may  require  moisture,  fungus,  and  corrosion 
prevention  techniques  in  design, 
manufacturing  and  surface  protection.  The 
advent  of  space  systems  and  the  Space 
Transportation  System  has  resulted  in  system 
application  requirements  for  contamination 
controls  which  far  exceed  other  system 
applications  in  complexity  and  mission  critical 
consequences.  Ultra-cleanliness  is  important, 
but  in  addition,  space  induced  and 
operationally  induced  contaminants  are 
critical  to  instruments  and  optics.  Outgassing 
of  materials,  volatile  condensable  materials, 
and  the  output  of  upper  stage  propulsion 
devices,  for  example,  are  some  of  the 
contaminants  which  have  to  be  controlled. 

Military  equipment  operates  under  the 
most  difficult  environments  imaginable. 
Extremes  of  temperature,  humidity,  sand, 
dust,  salt  spray,  rain,  all  tend  to  debilitate 
equipment  rapidly.  Without  proper 
resistance  or  protection,  much  equipment 
would  cease  to  function,  or  fail  to  function 
when  called  upon.  Protection  must  be 
provided  by  the  proper  specification  of 
materials,  covers  and  seals,  packaging, 
heating  and  cooling  devices,  or  other  design 
features  which  permit  extended  storage  and 
operation  throughout  the  range  of 
operational  environments.  Once  a 
qualification  unit  has  been  built,  it  must  be 


subjected  to  tests  which  encompass  the  full 
range  of  environmental  requirements,  plus 
a  margin  of  safety. 

433  Survivability/  Vulnerability 

Military  systems  are  vulnerable  to  the 
natural  environment  of  ground  or  space 
and  to  hostile  threats  from  ground  forces, 
air  forces,  and  anti-satellite  weapons  or 
nuclear  detonations.  Engineering 
specialists  in  survivability/vulnerability 
analyze  and  evaluate  these  natural  and 
induced  threats  in  the  C/E  phase  to 
determine  the  design  approach  and 
methods  required  to  meet  survivability 
requirements  in  the  system  specification. 

These  specialists  analyze  the 
vulnerability  of  the  system  in  terms  of  the 
capability  of  parts  and  materials,  protective 
measures,  system  architecture,  functions, 
and  controls.  Risks  and  alternatives  are 
assessed  in  tradeoff  studies  which  lead  to 
design  criteria  and  decisions  necessary  to 
reduce  vulnerability  and  improve 
survivability. 

A  survivability  issue  of  concern  to 
many  ground-,  air-,  and  space-based 
systems  is  that  of  electromagnetic  pulse 
(EMP),  one  of  a  number  of  nuclear 
weapons  effects  that  must  be  considered 
in  a  hardening  program.  EMP  is  a  burst 
of  radio  frequency  energy  whose  time 
signature  and  spatial  extent  depend  on 
the  height  and  location  of  a  nuclear  burst 
relative  to  the  point  of  observation.  The 
EMP  generated  wave  form  generally  bears 
little  resemblance  to  the  conducted 
transient  energy  that  arrives  at  a  point  of 
entry  to  electronic  equipment.  Hardening 
a  system  to  EMP  entails:  determining  the 
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external  threat  environment  and  the  coupling 
of  the  environment  to  the  system  to 
determine  the  energy  pulse  at  the  electronic 
equipment,  preparing  survivability 
specifications  for  that  equipment,  determining 
the  methods  of  protecting  that  equipment, 
selecting  the  analysis  and/or  test  techniques 
to  verify  the  hardening  approach,  and 
specifying  the  techniques  to  assure  and 
maintain  the  hardness  of  the  equipment  when 
produced  and  placed  in  service. 

4.4  HARDWARE/  SOFTWARE 

INTEGRATION 

Although  nor  normally  considered  an 
engineering  specialty,  hardware/  software 
integration  is  a  critical  and  unique  function 
in  ensuring  the  quality  of  a  product.  In 
order  to  provide  the  system  perspective 
required  to  successfully  accomplish  its  tasks, 
this  group  is  located  within  a  system 
engineering  department.  It  consists  of 
individuals  having  both  computer  hardware 
and  software  backgrounds  and  possessing  a 
good  understanding  of  system  configuration 
and  operation.  Systems  engineering  has  four 
basic  responsibilities  with  respect  to  software 
development:  1)  definition  of  the  system  level 
design  apr  2)  ana'  is  and  allocation  of 
system  acquirements  to  hardware  and 
software  co>  figuration  items,  3)  definition  of 
the  interfaces  between  hardware  and  software 
configuration  items,  and  4)  performance  of 
technical  reviews  and  audits. 

Systems  engineering  specifies  and 
approves  requirements  alloca*?'1  to  each 
computer  program  rnd  audits  ;  jrams  to 
verify  compliance  with  these  requirements. 
In  addition,  systems  engineering  provides 
interface  requirements  and  characteristic 
input  data  to  the  software  dt  pment 


personnel  and  verifies  the  successful 
integration  of  equipment  and  computer 
programs.  Systems  engineering  also 
performs  assessments  of  the  computer 
software  use,  of  computer  timing  and 
storage  resources,  and  the  status  of 
technical  progress. 

Systems  engineering  is  responsible  for 
conducting  design  reviews,  for  internal 
contractor  approval  of  software 
configuration  changes  (prior  to  submission 
for  government  approval),  and  for 
certifying  that  development  milestones  are 
satisfied.  Systems  engineering  also  works 
with  the  test  organization  to  verify  the 
proper  functioning  of  all  computer 
programs,  the  proper  integration  of  and 
nominal  execution  of  hardware  and 
software,  and  the  satisfactory 
demonstration  of  system  operation  over 
the  total  range  of  specified  operating 
conditions  (normal  and  abnormal). 

The  primary  instrument  of  hardware/ 
software  integration  is  the  hardware/ 
software  Interface  Control  Document 
(ICD).  The  ICD  is  required  to  specify  the 
functional  interface  between  the  computer 
program  product  and  any  equipment 
hardware  with  which  it  must  operate.  It 
is  often  true  that  supplier  documentation 
for  standard  computer  peripherals  and 
terminals  is  adequate  for  this  purpose. 
Conversely,  it  has  been  found  that 
performance  specifications  governing  the 
design  new  equipment  is  not  satisfactory 
for  use  as  a  functional  ICD.  The  purpose 
of  an  ICD  is  to  communicate  equipment 
interface  requirements  to  programmers  in 
terms  that  the  programmers  readily  and 
accurately  understand  and  to  require 
equipment  designers  to  consider  the 
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Figure  4-3 
Sample  Format  of  an 
Interface  Control  Document  (ICD) 


1.0  Scope:  Purpose  and  coverage 

2.0  Applicable  documents:  List  of  ail  documents  referenced  In  Sections  3  and  4 

3.0  Requirements 

3.1  Physical  Interface 

3.1.1  Mechanical  Requirements:  Envelope,  attachment,  obscuration,  alignment 

3.1.2  Master  Tooling 

3.1.3  Mass  Properties:  Weight,  moment-of-inertia,  center-of-gravity  location,  axes,  models  exchange 
(math/physical) 

3.2  Electronic  Interlace 

3.2.1  Command  Signals:  Format,  rates,  identification 

3.2.2  Data  signals:  Radio  frequency  characteristics,  format,  rate 

3.2.3  Telemetry  Signals:  Format,  clock,  identification,  recording 

3.3  Electrical  Interlace 

3.3.1  Electrical  Power  Type,  voltage,  power  profile,  protection 

3.3.2  Interface  Pin  Assignments 

3.3.3  Electromagnetic  Compatibility 

3.4  Hydraulic/Pneumatic  Interface:  Type,  flow  rate,  temperature,  pressure 

3.5  Software 

3.5.1  Data:  Inputs,  outputs,  rates,  accuracies 

3.5.2  Messages:  Format,  content,  storage 

3.5.3  Protocols:  Enable,  processing,  validation,  error  detection,  recovery 

3.6  Hardware/Software 

3.6.1  Interface:  Diagrams,  standards,  and  conventions 

3.6.2  Timing  and  Sequencing:  Control  and  logic,  relationships,  data  transfers,  input  sensing 

3.7  Environmental 

3.7.1  Structural:  Vibration,  shock,  acoustic,  loads,  dynamic  mode  shapes 

3.7.2  Thermal:  Temperature  range,  heating  rates,  heat  transfer  surfaces 

3.7.3  Magnetic:  Flux  density,  rate-of -change 

3.7.4  Radiation:  Type,  flux  density,  total  dose 

3.7.5  Ambient:  Pressure,  temperature,  containments 

3.7.6  Air  Conditioning:  Temperature,  flow  rates 

3.8  Safety 

3.9  Operational  Limitations 

4.0  Verification 

4.1  Quality  Assurance 

4.1.1  QA  Requirements 

4.1.2  ICD  Requirements  Verification:  Matrix 

4.2  Factory  Tests 

4.2.1  Facility  Requirements 

4.2JZ  Receiving  Inspection 

4.2.3  Installation  Requirements 

4.2.4  Test  Constraints 

4J2.5  Test  Sequence 

4.2.6  Shipping  Preparation 

4.3  Launch  Base  Tests 

Appendix:  Delivery  Schedules 
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impact  of  their  design  on  computer  programs. 

The  ICD  provides  an  exact  definition  of 
every  interface,  by  medium  and  by  function, 
including  input/output  control  codes,  data 
format,  polarity,  range,  units,  bit  weighing, 
frequency,  minimum  and  maximum  timing 
constraints,  legal/illegal  values,  and  accuracy, 
resolution,  and  significance.  A  sample  ICD 
format  is  provided  in  Figure  4-3.  Existing 
documentation  may  be  referenced  to  amplify 
explanations  of  the  effect  of  input/output 
operations  on  external  equipment.  Testing 
required  to  validate  the  interface  designs  is 
also  specified. 

4.5  INTERFACE  MANAGEMENT 

Precise  interface  definition  early  in  the 
program  is  essential  to  a  successful  and 
timely  development.  Using  functional 
analyses,  systems  engineering  develops 
functional  and  interface  requirements.  As 
the  total  system  is  "decomposed"  into 
functional  areas,  interfaces  between  the  areas 
appear.  These  may  be  physical  or 
operational  (or  both),  but  are  usually 
characterized  by  mechanical,  electrical,  or 
functional  data  parameters  or  procedures 
with  associated  data  requirements. 
Functional  and  performance  interface 
requirements  are  contained  in  the 
appropriate  segment  or  (Cl)  specifications. 
However,  the  design  implementation  of  these 
requirements  must  be  defined  to  all 
concerned  in  order  that  the  equipment,  when 
developed  and  integrated,  will  function 
satisfactorily. 

Interfaces  are  initially  defined  through 
the  use  of  functional  flow  block  diagrams 
and  functional  interface  input/output  charts. 
These  tools  primarily  define  the  flow  data. 


Once  the  functional  requirements  have 
been  completely  allocated  the  design 
implementation  of  the  interfaces  can 
proceed.  This  often  involves  a  number  of 
contractors  and  organizations  that  require 
close  coordination  by  system  engineering 
to  ensure  compatibility  of  all  interfaces. 
Interface  development  is  accomplished 
through  the  use  of  Interface  Control 
Working  Groups  (ICWGs).  These  ICWGs 
are  generally  organized  by  the  prime 
contractor  or  system  integrator,  if  the 
latter  is  used  on  the  program.  The 
government  PM  may  chair  these  groups, 
although  it  is  often  done  by  the  prime 
contractor,  with  the  government  being  co- 
chairman  to  ensure  resolution  of  any 
conflicts  that  may  arise  between  associate 
contractors.  The  ICWGs  may  be  divided 
into  interface  panels  that  handle 
specialized  areas  such  as  EMC,  computer 
resources,  and  test  planning.  The 
chairman  is  responsible  for  organizing  the 
groups,  ensuring  that  the  proper  specialties 
are  supported  by  individuals  who  have 
authority  to  commit  their  organizations  or 
who  are  charged  with  obtaining  their 
organizational  approval  for  ICDs  done  by 
the  ICWG.  The  chairman  also  prepares 
the  agendas,  conducts  the  meeting, 
prepares  minutes  of  the  meeting,  takes 
action  items,  and  assigns  responsibilities. 
The  prime  contractor  ensures  that  action 
items  are  completed  and  that  the 
responses  are  properly  documented  and 
transmitted  to  those  parties  involved  in  the 
design  implementation. 

The  results  of  this  activity  are 
documented  in  ICDs.  The  nature  of  the 
ICD  varies  considerably,  a  radio  frequency 
depending  on  the  interface  being 
documented.  It  can  be  a  physical,  an  RF, 
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or  an  operational  interface  as  illustrated  in 
Fig.  4.4  for  a  space  system.  Design  definition 
takes  the  forms  of  drawings,  schematics, 
function  lists,  data  format  diagrams, 
operational  procedures,  equations,  and  any 
other  data  required  by  the  designers  to 
completely  detail  their  design.  Electrical 
circuitry,  for  example,  is  usually  defined  to 
include  the  first  active  circuit  on  either  side 
of  the  interface.  The  ICD  does  not  duplicate 
the  specification,  rather  it  describes  the 
design  implementation  of  the  requirements 
in  the  specification. 

The  ICD  outline  is  prepared  by  the  prime 
contractor  and  portions  are  assigned  to 
parties  responsible  for  the  development  of 
the  interface.  Following  completion  of  the 
ICD,  it  is  signed  by  all  parties  involved  in 
the  interface  and  placed  under  configuration 
control.  The  ICD  then  has  the  same  status 
as  a  specification  in  that  it  represents  the 
baseline  configuration,  and  any  changes  must 
be  acted  upon  by  the  appropriate  CCB.  A 


number  of  CCBs  may  be  involved  in 
implementing  the  change,  including  those 
at  the  interfacing  contractors  and  the 
Government  Program  Office,  which  must 
approve  any  changes  to  the  segment 
interfaces. 


Integration  of  design  requirements  is 
part  of  the  system  engineering  process, 
whether  "designing*for" 
performance/constraints  associated  with 
traditional,  specialty,  or  product  oriented 
disciplines.  Each  discipline  "filter"  is 
integrated  within  the  basic  systems 
engineering  process  steps  of  functional 
analysis,  synthesis,  and  evaluation/  decision. 
All  disciplines  share  common  systems 
engineering  documentation  which  they  use 
to  express  requirements.  The  rigor 
imposed  by  a  formal  systems  engineering 
process  ensures  that  all  of  these  specialty 
disciplines  respond  to  requirements  in  a 


4.6  SUMMARY 


4-16 


timely,  integrated  manner  by  providing  a 
basic  model  for  the  activities  which  must  be 
accomplished  in  the  systems  engineering 
effort  and  an  integrated  data  system  for 
development  and  integration  of  system 
requirements. 

Figure  4-5  presents  some  general  task 
descriptions  for  an  engineering  process.  The 
figure  lists  tasks  that  engineering  personnel 
should  undertake  in  a  development  program 
using  systems  engineering  as  a  technical 
management  tool.  No  particular  discipline  is 
identified  by  each  task  since  most  disciplines 
must  1)  accomplish  planning,  2)  identify  and 
allocate  functional  requirements,  3) 
participate  in  trade  studies,  4)  provide  inputs 
to  the  documentation,  and  5)  participate  in 
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and  trade-offs  by  the  systems  engineering 
function,  and  must  be  uniquely  described 
in  the  SEMP  as  well  as  all  associated 
"specialty"  plans.  Often  integration  is 
partially  achieved  using  common  data 
formats.  Figure  4-5  illustrates  how  systems 
engineering  documentation  is  used  to 
support  tasks  which  contribute  to  the 
integration  of  engineering  disciplines  in 
each  life  cycle  phase.  In  this  figure,  where 
a  relationship  between  a  task  and  a 
documentation  form  is  indicated,  a 
potential  use  of  the  documentation  in 
support  of  the  task  is  suggested.  Where 
no  relationship  is  shown,  no  specific  form 
is  related  to  the  integration  task. 
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Figure  4-5  (1  of  4) 
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Figure  4-5  (2  of  4) 
Principal  systen  integration  Tasks 
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Figure  4-5  <3  of  4) 
Principal  System  integration  Tasks 
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Figure  4-5  <4  of  4) 
Principal  System  Integration  Tasks 
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CHAPTER  5 


THE  SYSTEMS  ENGINEERING  PROCESS 


5.1  INTRODUCTION 

This  chapter  summarizes  the  activities 
which  occur  during  the  systems  engineering 
process  and  describes  the  basic  steps  which 
comprise  the  process:  functional  analysis 
(function  identification,  and  requirements 
identification  and  allocation),  synthesis, 
evaluation  and  decision,  and  description  of 
the  system  elements.  The  elements  which 
define  a  system  are  equipment  (hardware), 
software,  facilities,  personnel,  and  procedural 
data. 

The  systems  engineering  process  is  an 
iterative  process  applied  throughout  the 
acquisition  life  cycle.  The  process  itself  leads 
to  a  well  defined,  completely  documented, 
and  optimally  balanced  system.  It  does  not 
produce  the  actual  system  itself,  but  rather, 
it  produces  the  complete  set  of 
documentation,  tailored  to  the  needs  of  a 
specific  program,  which  fully  describes  the 
system  to  be  developed  and  produced.  Each 
program’s  systems  engineering  process, 
developed  through  tailoring  and/or  adding 
supplemental  requirements,  must  meet 
certain  general  criteria.  Although  not 
complete,  the  following  guidelines  should  be 
considered  in  one’s  approach  to  the  basic 
process  [I]: 

a.  System  and  subsystem  (configuration 
item)  requirements  shall  be  consistent, 
correlatable,  and  tiaceable  both  within  data 


produced  as  basic  documentation  (e.g., 
Functional  Flow  Block  Diagram  (FFBD), 
Requirements  Allocation  Sheet  (RAS),  and 
Time  Line  Sheet  (TLS))  and  as  related 
documentation  (e.g.,  work  breakdown 
structure  (WBS)  and  Logistic  Support 
Analysis  Record  (LSAR)). 

b.  The  concept  of  minimum 
documentation  shall  be  evident. 

c.  Acquisition  and  ownership  cost  shall 
be  an  integral  part  of  the  evaluation  and 
decision  process. 

d.  Baselines  which  meet  the  general 
guidance  of  MIL-STD-490A  [2]  shall  be 
established  progressively  as  an  integral  part 
of  the  systems  engineering  process. 

e.  The  systems  engineering  process  shall 
result  in  a  design  that  is  complete,  at  a  given 
level  of  detail,  from  a  total  system  element 
viewpoint. 

f.  The  process  shall  provide  for  the 
timely  and  appropriate  integration  of 
mainstream  engineering  with  engineering 
specialties  such  as  reliability,  maintainability, 
human  factors  engineering,  safety,  integrated 
logistic  support  (ILS),  environmental 
assessments,  and  producibility  to  ensure  their 
influence  on  system  design. 

g.  The  process  shall  provide  for 
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continuing  prediction  and  demonstration  of 
the  anticipated  or  actual  achievement  of  the 
primary  technical  objectives  of  the  system. 
Problems  and  risk  areas  shall  be  identified  in 
a  timely  manner. 

h.  Formal  technical  reviews  and  audits 
shall  be  an  integral  part  of  the  systems 
engineering  process.  The  requirements  for 
these  reviews  and  audits  are  given  in 
MIL-STD-1521B  [3]. 

i.  The  systems  engineering  process  shall 
be  responsive  to  change.  The  impact  of 
changes  to  system  and/or  program 
requirements  must  be  traceable  to  the  lowest 
level  of  related  documentation  in  a  timely 
manner. 

j.  Significant  engineering  decisions  shall 
be  traceable  to  the  systems  engineering 
activities  and  associated  documentation  upon 
which  they  were  based. 


5.2  THE  BASIC  SYSTEMS  ENGINEERING 

PROCESS 

Figure  5-1  presents  an  overview  of  the 
four  basic  steps  of  the  systems  engineering 
process. 

5.2.1  Functional  Analysis 

Every  engineering  effort  must  begin  with 
a  statement  of  a  perceived  need.  At  the 
beginning  of  a  DOD  acquisition  effort,  this 
statement  will  be  in  the  form  of  a  system 
requirement  document,  usually  developed 
through  a  Mission  Area  Analysis  of 
anticipated  threats.  Once  the  purpose  of  the 
system  is  known,  the  functional  analysis 
activity  identifies  what  essential  functions  the 
system  must  perform.  In  order  to  accomplish 
this,  functional  analysis  is  composed  of  two 
primary  process  segments:  1)  functional 
identification  and  2)  requirements 
identification  and  allocation  (functional 


Figure  5-1 

The  Systems  Engineering  Process 
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performance  requirements  analysis).  It 
answers  the  "what"  and  "why"  questions 
relative  to  system  design.  Further  detail  is 
provided  in  Chapter  6  of  this  guide. 

The  basic  analytical  tool  for  functional 
identification  is  the  Functional  Flow  Block 
Diagram  (FFBD),  showing  logical  sequences 
and  relationships  of  operational  and  support 
functions  at  the  system  level.  Specific 
functions  will  vary  from  system  to  system  and 
will  be  traceable  to  mission  requirements  and 
objectives.  Maintenance  flow  diagrams  which 
depict  general  maintenance  and  support 
concepts  will  lead  to  analysis  of  requirements 
on  an  end  item/equipment  basis.  At  this 
level,  since  functions  are  more  standardized, 
functional  identification  is  often  accomplished 
using  the  End  Item  Maintenance  Sheet 
(EIMS)  or  Logistic  Support  Analysis  Record 
(LSAR).  Similarly,  detailed  test  requirements 
are  identified  using  the  Test  Requirements 
Sheet  (TRS),  and  productivity  requirements 
are  identified  using  the  Production  Sheet 
(PS). 

It  should  be  kept  in  mind  that  the 
systems  engineering  process  is  always 
iterative.  Each  acquisition  phase  will  involve 
functional  analysis  to  progressively  more 
detail.  For  example,  during  the  Concept 
Exploration/  Definition  (C/E)  phase,  analysis 
of  support  functions  will  concentrate  on 
Maintenance  FFBDs,  which  will  support  the 
establishment  of  gross  maintenance  concepts. 
During  Full  Scale  Development  (FSD), 
emphasis  will  shift  to  detailed  analysis  of  the 
maintenance  requirements  of  specific 
equipment  using  the  EIMS  or  LSAR. 

The  Requirements  Allocation  Sheet 


(RAS)  is  used  as  the  primary  analytical  tool 
for  requirements  identification  and  allocation, 
or  functional  performance  requirements 
analysis  as  it  often  is  referred  to,  in 
conjunction  with  FFBDs  and  special  purpose 
documents  such  as  EIMSs,  TRSs,  and  PSs. 
The  RAS  serves  three  purposes  in 
documenting  the  systems  engineering  process: 
1)  initially,  it  is  used  to  record  the 
performance  requirements  established  for 
each  function;  2)  during  synthesis,  it  is  used 
to  show  the  allocation  of  the  functional 
performance  requirements  to  individual 
system  elements  or  a  combination  of 
elements;  and  3)  following  evaluation  and 
decision,  the  RAS  provides  the  functionally 
oriented  data  required  in  the  description  of 
the  system  elements. 

The  Time  Line  Sheet  (TLS)  is  used  to 
perform  and  record  the  analysis  of 
time-critical  functions  and  functional 
sequences.  In  performing  time  requirements 
analysis  for  complex  functional  sequences, 
additional  tools,  such  as  mathematical  models 
or  computer  simulations,  may  be  needed. 
Time  requirements  analysis  is  performed  in 
any  or  all  of  the  functional  cycles  of  the 
process  to  determine  whether  time  is  a 
critical  factor.  The  TLS  complements  the 
FFBD  in  its  ability  to  show  a  lower  level  of 
detail,  as  well  as  to  illustrate  the  impact  of 
concurrent  functions  within  a  given  sequence. 
TLSs  are  used  to  support  the  development 
of  design  requirements  for  the  operation, 
test,  and  maintenance  functions.  They 
identify  time-critical  functions  and  depict  the 
concurrency,  overlap,  and  sequential 
relationship  of  functions  and  related  tasks. 
Time-critical  functions  are  those  that  affect 
reaction  time,  down  time,  or  availability. 
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5.2.2  Synthesis 

Synthesis  supplies  the  "how”  answers  to 
the  "what"  outputs  of  functional  analysis. 
Further  detail  on  synthesis  techniques  is 
presented  in  Chapter  7  of  this  guide. 

Two  documentation  tools  accomplish 
and  record  the  synthesis  of  design  approaches 
or  alternative  approaches.  The  Concept 
Description  Sheet  (CDS)  is  used  to  collect 
the  perf  an,  e  requirements  and 
constraim  .  ,ited  by  functional 

analysis,  '  ,-.rp.;y  to  an  individual  subsystem 
or  e  .  *  >.m.  The  CDS  also  describes  at  the 
gross  level  a  design  approach  for  meeting  the 
requirements.  The  Schematic  Block  Diagram 
(SBD)  is  used  to  develop  and  portray  the 
conceptual  schematic  arrangement  of  system 
elements  to  meet  system  and/or  subsystem 
requirements.  The  CDS  and  SBD  are  both 
applicable  to  all  acquisition  phases  and 
provide  the  basis  for  development  of  the 
descriptions  of  system  elements. 

5.2.3  Evaluation  and  Decision 

Since  program  risk  and  cost  are 
dependent  on  practical  trade-offs  between 
stated  operating  requirements  and 
engineering  design,  continual  evaluations  and 
decisions  must  be  made  not  only  at  the 
beginning  of  the  program  but  throughout  the 
design  and  development  activity.  Several 
approaches  to  the  evaluation  and  decision 
process  is  presented  in  Chapter  8  of  this 
guide. 

The  Trade  Study  Report  (TSR)  is 
used  to  summarize  and  correlate 
characteristics  of  alternative  solutions  to  the 
requirements  and  constraints  which  establish 


the  selection  criteria  for  a  specific  trade  study 
area.  The  report  also  documents  the 
rationale  used  in  the  decision  process  and 
should  present  risk  assessment  and  risk 
avoidance  considerations.  Other  tools,  such 
as  analytical  or  mathematical  models  or 
computer  simulations,  may  be  needed  and 
used  in  accomplishing  the  evaluation  and 
decision  process. 

5.2.4  Description  of  System  Elements 

All  systems  can  be  defined  by  a  set  of 
interacting  system  elements  which  fall  into 
five  categories:  1)  equipment  (hardware),  2) 
software,  3)  facilities,  4)  personnel,  and  5) 
procedural  data.  Two  documentation  forms 
are  used  to  describe  these  system  elements: 
1)  the  Design  Sheet  (DS)  and  2)  the  Facility 
Interface  Sheet  (FIS).  The  DS  is  used  to 
establish  and  describe  the  performance, 
design,  and  test  requirements  for  equipment 
end  items,  critical  components,  and  computer 
software  programs.  The  FIS  is  used  to 
identify  the  environmental  requirements  and 
interface  design  requirements  imposed  upon 
facilities  by  the  functional  and  design 
characteristics  of  equipment  end  items.  The 
DS  and  FIS  provide  the  basis  for  the  formal 
identification  required  for  configuration 
management. 

S3  DOCUMENTATION 

The  systems  engineering  process  produces 
the  basic  and  special  purpose  documentation 
which  controls  the  evolutionary  development 
of  the  system.  Figure  5-2  correlates  the 
particular  documentation  associated  with  each 
step  of  the  systems  engineering  process. 

The  systems  engineering  process  itself 
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Figure  5-2 

Basic  and  Special  Purpose  Documentation  for  Systems  Engineering 


IS  CARRIED  TO  THE  LEVEL  DEC  £0  FOB  THE  SELECTED  LEVEL  OF  ENOMEERMO  TO  JDENTFY,  DESCRIBE  AND  SPECIFY 


doesn’t  actually  produce  the  system,  but 
produces  the  documentation  necessary  to 
define,  design,  develop,  and  test  the  system. 
As  such,  a  variety  of  engineering  and 
planning  documentation  is  required 
throughout  the  acquisition  cycle  and  systems 
engineering  is  the  vehicle  used  to  produce 
that  documentation. 

Numerous  plans  are  prepared  to 
define  which  technical  activities  will  be 
conducted.  They  address  the  integration  of 
engineering  specialties  requirements, 
"design-for"  requirements,  and  organic  >  ;  ' 

resource  requirements,  and  discuss  bo- 
progress  toward  system  level  gc?N  \->u 
measured.  The  Systems  fcr»g;neering 
Management  Plan  (SEMP)  is  the  key 
planning  document  which  reflects  these 
requirements.  Contractor  compliance  with 
these  plans  is  monitored  by  government 
organizations  to  ensure  that  standard  policies 
and  procedures  in  the  area  of  systems 
engineering  are  employed.  Additionally, 
specifications  are  prepared  as  part  of  the 
systems  engineering  process  to  form  the  basis 
for  the  design  and  development  effort.  The 
top  level  specification  (system  or  segment)  is 
normally  approved  and  draft  lower  level 
specifications  (configuration  items)  are 
developed  reflecting  allocated  system 
requirements  to  lower  level  components  or 
subsystems,  which  designers  and 
subcontractors  translate  into  hardware  and 
software  production  plans. 

In  order  to  provide  a  continuing 
assessment  of  the  system’s  capability  to  meet 
performance  requirements,  the  systems 
engineering  organization  prepares  technical 
review  data  packages,  technical  performance 
measurement  (TPM)  reports,  analysis  and 


simulation  reports,  and  other  documentation 
which  normally  includes:  Interface  Control 
Documents/  Drawings,  Trade  Study  Reports/ 
Directions,  Risk  Analysis  Management  Plan, 
Survivability/  Hardness  Plan,  Mission  Area 
Analysis  Reports,  Functional  Analysis 
Documentation,  Reliability  Plan, 
Maintainability  Plan,  Safety/  Hazard  Analysis 
Plan,  Human  Engineering  Plan,  Integrated 
Logistic  Support  Plan  (ILSP), 
Electromagnetic  Compatibility/  Interference 
Control  (EMC/  EMI)  Plan,  Test  and 
Evaluation  Master  Plan  (TEMP),  Mission 
Support  Plan,  and  Production  Engineering 
ran. 

5.4  SUMMARY 

The  systems  engineering  process  is  one 
approach  to  providing  disciplined  engineering 
during  all  acquisition  phases.  Although 
current  application  of  the  process  has  focused 
on  C/E,  D/V,  and  FSD,  systems  engineering 
process  techniques  and  principles  are  equally 
applicable  to  the  analysis  and  definition  of 
production  requirements.  DoD  4257.7-M  [4J 
enables  the  Program  Manager  to  review  and 
evaluate  the  products  of  the  systems 
engineering  process  from  the  standpoint  of 
managing  risk  during  the  transition  from 
development  to  production.  Chapter  15  of 
this  guide  fully  describes  risk  management  as 
it  relates  to  the  systems  engineering  process. 

The  systems  engineering  process  provides 
the  logic  and  timing  for  a  disciplined 
approach,  with  certain  internal  assurances  of 
technical  integrity  such  as  traceability. 
Technical  integrity  ensures  that  the  design 
requirements  for  the  system  elements  reflect 
the  functional  performance  requirements,  that 
all  functional  performance  requirements  are 
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satisfied  by  the  combined  system  elements, 
and  that  such  requirements  are  optimized 
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CHAPTER  6 


FUNCTIONAL  ANALYSIS 


6.1  INTRODUCTION 

Functional  analysis,  as  the  first  step  in 
the  systems  engineering  process,  defines  a 
baseline  of  functions  and  function 
performance  requirements  which  must  be 
met  in  order  to  adequately  accomplish  the 
operation,  support,  test,  and  production 
requirements  of  the  system.  Functional 
analysis  begins  with  the  identification  of  top 
level  functions  and  ends  with  the  allocation 
of  those  functions  to  lower  level  elements 
within  the  system.  This  effort  should  be 
influenced  by  the  synthesis  (the  second  step 
of  the  systems  engineering  process)  of  system 
elements  in  verifying  their  capability  to 
accomplish  the  allocated  requirements.  In 
other  words,  functional  analysis  and  synthesis 
should  be  performed  in  concert  because  the 
synthesis  must  be  responsive  to  functional 
requirements.  These  functional  requirements 
provide  a  common  basis  for  the  selection 
and  design  criteria  for  system  elements  and 
identify  areas  where  trade-offs  between  input 
requirements  and  engineering  development 
require  future  consideration. 

Functional  analysis  is  a  method  for 
analyzing  performance  requirements  and 
dividing  them  into  discrete  tasks  or  activities. 
It  involves  the  identification  and 
decomposition  of  the  primary  system 
functions  into  subfunctions  at  ever  increasing 
levels  of  detail.  It  supports  mission  analysis 
in  defining  functional  areas,  sequences,  and 
interfaces  and  is  also  used  by  engineering 


specialists  to  develop  requirements  for 
equipment,  software,  personnel,  and 
operational  procedures  to  complete 
implementation  and  deployment  of  the 
system.  There  are  two  basic  activities 
associated  with  functional  analysis:  1) 
functional  identification  and  2)  requirements 
allocation.  This  chapter  describes  a  number 
of  tools  used  to  accomplish  these  activities 
and  the  means  of  documenting  the  effort;  it 
is  not  meant  to  provide  a  detailed  guide  for 
undertaking  a  functional  analysis. 

6.2  APPROACH 

A  function  is  a  characteristic  action  to  be 
accomplished  by  one  of  the  system  elements 
of  equipment  (hardware),  software,  facilities, 
personnel,  procedural  data,  or  any 
combination  thereof.  Functional 
identification  and  decomposition  can  be 
performed  with  respect  to  logical  groupings, 
time  ordering,  data  flow,  control  flow,  or 
some  other  criterion.  The  stepwise 
decomposition  of  a  system  can  be  viewed  as 
a  top  down  approach  to  problem  solving. 

This  top  down  approach  is  illustrated  in 
Figure  6-1,  which  shows  a  system  being 
separated  into  functional  areas  or  segments. 
Each  functional  area  satisfies  an  allocated 
portion  of  the  basic  system  functions. 
Collectively,  these  areas  constitute  a  complete 
system  description  at  each  level.  When  these 
segments  are  separated,  as  they  actually  may 
be  in  a  physical  sense,  required  interface 
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connections  are  exposed.  As  the  functions 
are  decomposed  to  the  next  lower  level 
(element),  the  number  of  functions 
(requirements)  greatly  increases,  each  with  its 
own  interfaces.  This  process  continues  until 
the  lowest  level  is  reached  and  where 
discrete  tasks  (such  as  compute  range)  can 
be  defined  and  satisfied.  Note  that 
traceability  is  maintained  throughout  by  a 
decimal  numbering  system.  Although  Figure 
6-1  shows  each  function  generating  an  equal 
number  of  subfunctions,  this  is  seldom  the 
case  for  real  systems. 


One  of  the  most  important  advantages  of 
top  down  development  is  that  the  most 
difficult  design  area  can  be  attacked  first 
throughout  its  total  hierarchy;  e.g.,  in  Figure 
6-1,  doing  all  of  3  (3.1,  3.2,  3.3;  3.1.1,  3.1.2, 
3.1.3,  etc.)  at  the  start  of  the  development  to 
reduce  risk.  It  should  be  noted  that  top 
down  development  is  not  the  same  as  the 
top  down  division  of  program  effort  that  is 
employed  in  developing  the  work  breakdown 
structure  (WBS).  However,  there  appears  to 
be  a  similarity  between  the  WBS  and  top 
down  development  methodology,  since  the 


Figure  6-1 

Top-Down  Approach  to  Functional  Decomposition 
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hardware  items  and  software  modules  that 
will  perform  the  discrete  and  interface  tasks 
identified  through  functional  analysis  must  be 
WBS  elements. 

63  FUNCTIONAL  IDENTIFICATION 

System  requirements  are  analyzed  to 
identify  those  functions  which  must  be 
performed  to  satisfy  the  objectives  of  each 
functional  area.  Each  function  is  identified 
and  described  in  terms  of  inputs,  outputs, 
and  interface  requirements  from  top  down  so 
that  subfunctions  are  recognized  as  part  of 
larger  functional  areas.  Functions  are 
arranged  in  a  logical  sequence  so  that  any 
specified  operational  usage  of  the  system  can 
be  traced  in  an  end-to-end  path.  Although 
there  are  many  tools  available,  functional 
identification  is  accomplished  primarily 
through  the  use  of  1)  functional  flow  block 
diagrams  (FFBDs)  to  depict  task  sequences 
and  relationships,  2)  N2  diagrams  to  develop 
data  interfaces,  and  3)  time  line  analyses  to 
depict  the  time  sequence  of  time  critical 
functions. 

63.1  Functional  Flow  Block  Diagrams 

The  purpose  of  the  FFBD  is  to  indicate 
the  sequential  relationship  of  all  functions 
that  must  be  accomplished  by  a  system. 
FFBDs  depict  the  time  sequence  of 
functional  events.  That  is,  each  function 
(represented  by  a  block)  occurs  following  the 
preceding  function.  Some  functions  may  be 
performed  in  parallel,  or  alternate  paths  may 
be  taken.  The  duration  of  the  function  and 
the  time  between  functions  is  not  shown  but 
may  vary  from  a  fraction  of  a  second  to 
many  weeks.  The  FFBDs  are  function 
oriented,  not  equipment  oriented.  In  other 


words,  they  identify  "what"  must  happen  and 
do  not  assume  a  particular  answer  to  "how" 
a  function  will  be  performed. 

FFBDs  are  developed  in  a  series  of 
levels.  FFBDs  show  the  same  tasks 
identified  through  functional  decomposition 
(such  as  those  portrayed  in  Figure  6-1)  and 
display  them  in  their  logical,  sequential 
relationship.  For  example,  the  entire  flight 
mission  of  a  spacecraft  can  be  defined  in  a 
top  level  FFBD,  as  shown  in  Figure  6-2. 
Note  that  the  numbers  in  Figure  6-1 
correspond  to  the  element  numbers  in  Figure 
6-2,  Each  block  in  the  first  level  diagram 
can  then  be  expanded  to  a  series  of 
functions,  as  shown  in  the  second  level 
diagram  for  "perform  mission  operations." 
Note  that  the  diagram  shows  both  input 
(transfer  to  operational  orbit)  and  output 
(transfer  to  space  transportation  system 
orbit),  thus  initiating  the  interface 
identification  and  control  process.  Each 
block  in  the  second  level  diagram  can  be 
progressively  developed  into  a  series  of 
functions,  as  shown  in  the  third  level  diagram 
on  Figure  6-2.  These  diagrams  are  used  both 
to  develop  requirements  and  to  identify 
profitable  trade  studies.  For  example,  does 
the  spacecraft  antenna  acquire  the  tracking 
and  data  relay  satellite  (TDRS)  only  when 
the  payload  data  are  to  be  transmitted,  or 
does  it  tr*ck  TDRS  continually  to  allow  for 
the  recepuon  of  emergency  commands  or 
transmission  of  emergency  data?  The  FFBD 
also  incorporates  alternate  and  contingency 
operations,  which  improve  the  probability  of 
mission  success.  The  flow  diagram  provides 
an  understanding  of  total  operation  of  the 
system,  serves  as  a  basis  for  development  of 
operational  and  contingency  procedures,  and 
pinpoints  areas  where  changes  in  operational 
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Figure  6-2 

Development  of  Functional  Flow  Block  Diagrams 
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procedures  could  simplify  the  overall  system 
operation.  In  certain  cases,  alternate  FFBDs 
may  be  used  to  represent  various  means  of 
satisfying  a  particular  function  until  data  are 
acquired,  which  permits  selection  among  the 
alternatives. 

6_3.2  N2  Diagrams 

The  N2  diagram  has  been  used 
extensively  to  develop  data  interfaces, 
primarily  in  the  software  areas;  however,  it 
can  also  be  used  to  develop  hardware 
interfaces.  The  basic  N2  chart  is  shown  in 
Figure  6-3.  The  system  functions  are  placed 


on  the  diagonal  and  the  remainder  of  the 
squares  in  the  N  x  N  matrix  represent  the 
interface  inputs  and  outputs.  Where  a  blank 
appears,  there  is  no  interface  between  the 
respective  functions.  Data  flows  in  a 
clockwise  direction  between  functions;  e.g., 
the  symbol  F;  F^  indicates  data  flowing 
from  function  F;  to  function  F2.  The  data 
being  transmitted  can  be  defined  in  the 
appropriate  squares.  Alternatively,  the  use  of 
circles  and  numbers  permits  a  separate  listing 
of  the  data  interfaces  as  shown  in  Figure  6- 
4.  The  clockwise  flow  of  data  between 
functions  that  have  a  feedback  loop  can  be 
illustrated  by  a  hrger  circle  called  a  control 
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loop.  The  identification  of  a  critical  function 
is  also  shown  in  Figure  6-4  where  function 
F^  has  a  number  of  inputs  and  outputs  to  all 
other  functions  in  the  upper  module.  A 
simple  flow  of  interface  data  exists  between 
the  upper  and  lower  modules  at  functions  F7 
and  F$.  The  lower  module  has  complex 
interaction  between  its  functions.  The  N2 
chart  can  be  taken  down  into  sucessively 
lower  levels  to  the  hardware  and  software 
component  functional  levels.  In  addition  to 
defining  the  data  that  must  be  supplied 
across  the  interface,  The  N2  chart  can 
pinpoint  areas  where  conflicts  could  arise. 


633  Time  Line  Analysis 

Time  line  analysis  adds  consideration  of 
functional  durations  and  is  used  to  support 
the  development  of  design  requirements  for 
operation,  test,  and  maintenance  functions. 
The  time  line  sheet  (TLS)  is  used  to  perform 
and  record  the  analysis  of  time  critical 
functions  and  functional  sequences. 
Additional  tools  such  as  mathematical  models 
and  computer  simulations  may  be  necessary. 
Time  line  analysis  is  performed  on  those 
areas  where  time  is  critical  to  the  mission 
success,  safety,  utilization  of  resources, 


Figure  6-4 

N2  Chart  Key  Features 

(From  "The  N2  Chart",  R.  Lano,  copyright  1977  TRW  Inc.) 


Figure  6-5 

Flight  Mission  Time  Lines 


minimization  of  down  time,  and/or  increasing 
availability.  Not  all  functional  sequences 
require  time  line  analysis,  only  those  in  which 
time  is  a  critical  factor.  The  following  areas 
are  often  categorized  as  time  critical:  1) 
functions  affecting  system  reaction  time,  2) 
mission  turn  around  time,  3)  time  countdown 
activities,  and  4)  functions  requiring  time  line 
analysis  to  determine  optimum  equipment 
and/or  personnel  utilization,  n  example  of 
a  high  level  TIiS  for  a  space  program  is 
shown  in  Figure  6-5. 

For  time  critical  function  sequences,  the 
time  requirements  are  specified  with 
associated  tolerances.  Time  line  analyses  play 


an  important  role  in  the  trade-off  process 
between  man  and  machine.  The  decisions 
between  automatic  and  manual  methods  will 
be  made  and  will  determine  what  times  are 
allocated  to  what  subfunctions.  In  addition  to 
defining  subsystem/  component  time 
requirements,  time  line  analysis  can  be  used 
to  develop  trade  studies  in  areas  other  than 
time  considerations;  e.g.,  should  the 
spacecraft  location  be  determined  by  the 
ground  network  or  by  onboard  computation 
using  navigation  satellite  inputs?  Figure  6-6 
is  an  example  of  a  maintenance  TLS  which 
illustrates  that  availability  of  an  item 
(distiller)  is  dependent  upon  the  completion 
of  numerous  maintenance  tasks  accomplished 
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concurrently.  Furthermore,  it  illustrates  the 
traceabilty  to  higher  level  requirements  by 
referencing  the  appropriate  FFBD  and 
requirement  allocation  sheet  (RAS). 

6.4  REQUIREMENTS  ALLOCATION 

Requirements  allocation  is  the  further 
decomposition  of  system  level  requirements 
until  a  level  is  reached  at  which  a  specific 
hardware  item  or  software  routine  can  fulfill 
the  needed  functional/  performance 
requirements.  It  is  the  logical  extension  of 
the  initial  functional  identification  and  an 
integral  part  of  any  functional  analysis  effort. 
Some  straightforward  allocat  ion  of  functional 
requirements  can  be  made,  but  the  procedure 
may  involve  the  use  of  supporting  analyses 
and  simulations  to  allocate  system  level 


requirements.  An  example  of  the  need  for 
additional  analysis  is  the  allocation  of 
availability  goals  to  configuration  items. 
These  goals  can  only  be  expressed  as 
maintainability  and  reliability  requirements. 
Allocations  and  trade  studies  will  be  made 
by  these  parameters  (maintainability  and 
reliability),  but  only  in  conjunction  with 
analytical  and/or  computer  simulation  to 
ascertain  the  impact  of  a  given  set  of 
allocations  on  system  availability. 

A  critical  aspect  of  the  documented 
systems  engineering  process  is  the  need  to 
provide  traceability.  Traceability  is  the  ability 
to  move  to  progressively  higher  or  lower 
levels  of  analysis  documentation.  It  includes 
tracking  the  allocation  design  (and  technical 
program)  requirements  through  the  WBS 


Figure  6-6 

sample  Maintenance  Time  Line  Sheet 
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between  the  system  level  and  the  lowest  level 
of  assembly.  Traceability  of  systems 
engineering  documentation  ensures  that  the 
impact  of  changes  to  requirements  at  any 
level  of  the  systems  engineering 
.documentation  or  program  specifications  can 
be  reviewed  for  impact  on  the  total  system, 
and  that  the  rationale  can  be  reviewed 
without  the  need  to  reconstruct  analyses.  The 
concept  of  traceability  is  graphically 
illustrated  in  Figure  6-7. 

The  systems  engineering  documentation 
described  in  Chapter  5  of  this  guide  provides 
the  audit  trail  for  allocation  traceability. 
Figure  6-7  portrays  an  example  of  the 
mechanics  used  to  provide  traceability  within 
the  systems  engineering  documentation. 


Prior  to  synthesis,  all  requirements  and 
other  analytical  data  are  oriented  to  functions 
and  are  identified  by  the  function  number  to 
which  they  pertain.  During  synthesis,  system 
elements  or  candidate  elements  are  identified 
to  satisfy  the  functional  performance 
requirements.  After  synthesis,  all 
requirements  and  other  design  data  are 
oriented  to  system  elements,  and  are 
identified  by  the  appropriate  configuration 
item  (Cl)  number  (or  similar  identification 
for  other  elements).  Any  program’s  tailored 
or  contractor  format  documentation  must 
have  similar  capability. 

The  end  result  of  the  requirements 
allocation  process  is  the  development  of  the 
system  level  (Type  A)  specification  and 


Figure  6-7 

Traceability  in  Systems  Engineering  Documentation 


development  (Type  B)  specifications  that 
represent  the  product  of  the  systems 
engineering  process.  The  importance  of  a 
well  documented  requirements  allocation 
cannot  be  overemphasized;  it  forms  the 
foundation  of  the  overall  systems  engineering 
effort. 

The  flowdown  of  system  requirements  to 
lower  levels  is  based  upon  the  mission  area 
analysis  (MAA)  and  system  level  FFBDs. 
The  initial  source  of  the  requirements  is  the 
Justification  for  Major  System  New  Start 
(JMSNS).  The  JMSNS  defines  the  mission 
need,  identifies  boundary  conditions,  and 
outlines  the  initial  acquisition  strategy.  The 
JMSNS  is  produced  through  government 
mission  area  analysis  studies  which  precede 
the  Concept  Exploration/  Definition  (C/E) 
phase.  These  requirements  are  often 
summarized  for  the  contractors  in  a  system 
requirements  document.  During  the  C/E  and 
Concept  Demonstration/  Validation  (D/V) 
phases,  requirements  from  this  document  are 
further  analyzed  by  each  contractor  through 
the  systems  engineering  process  and 
incorporated  into  the  system  specification  and 
flowed  down  to  the  lower  level  development 
specifications. 

Systems  engineering  documentation  for 
requirements  allocation  provides  a  vehicle  for 
system  specialists  to  document  initial 
allocation  of  technical  parameters  such  as 
weight,  reliability,  system  error,  and 
maintainability  prior  to  the  availability  of  the 
development  specification  for  configuration 
items.  The  systems  engineering 
documentation  integrates  technical  budgets, 
provides  requirements  traceability,  and 
ensures  a  consistent  consideration  of  ail 
constraining  parameters  at  all  levels  of 


evaluation  and  decision. 

The  systems  engineering  process  should 
avoid  indiscriminate  decomposition  and 
non-time/  resource  bounded  analysis. 
Specific  goals  related  to  the  depth  and  scope 
of  identification  and  allocation  of 
requirements  should  be  clearly  stated  in  the 
Systems  Engineering  Management  Plan 
(SEMP).  Detailed  guidance  for  tailoring  the 
systems  engineering  process  and 
documentation  is  provided  in  MIL-STD-499A 
[1].  Often  these  tailoring  goals  can  be  best 
stated  in  the  SEMP  as  a  requirement  for  the 
support  that  systems  engineering  analysis  and 
documentation  will  be  expected  to  provide 
for  the  formal  technical  review  process. 

The  requirements  analysis  is  usually 
considered  complete  when  a  further 
decomposition  of  functions  or  tasks  does  not 
result  in  additional  requirements  for 
equipment,  facilities,  software,  or  personnel. 
Decomposition  of  functions  should  only  be 
carried  out  to  the  point  where  further 
analysis  will  not  yield  additional  performance 
requirements  which  must  bear  on  synthesized 
and/or  selected  system  elements.  Once 
configuration  items  have  been  identified  for 
candidate  systems,  requirements  are  carried 
to  further  levels  of  detail  in  the  development 
specifications),  engineering  drawings,  and 
related  engineering  documentation.  The 
transition  from  Type  B  to  product  (Type  C) 
specifications  would  normally  begin  following 
Preliminary  Design  Review  (PDR)  and 
should  be  essentially  complete  by  Critical 
Design  Review  (CDR). 

6.4.1  Requirements  Allocation  Sheet 

The  primary  documentation  used  for 
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requirements  identification  and  allocation  is 
the  requirements  allocation  sheet  (RAS). 
The  RAS,  illustrated  in  Figure  6-8,  is  initially 
used  to  document  the  performance 
requirements  for  each  function  depicted  in 
the  FFBD.  During  functional  analysis,  as 
lower  level  functions  are  identified,  the  RAS 
rather  than  the  FFBD  is  generally  used  to 
document  functions  which  are  not  time 
critical,  to  avoid  many  levels  of  FFBD 
graphic  expansion.  An  RAS  will  usually  be 
developed  for  each  function  block;  however, 
there  are  cases  where  a  group  of  blocks 
having  closely  related  functions  may  be 
analyzed  by  a  combined  RAS.  Performance 
requirements  are  stated  in  terms  of  1)  the 
purpose  of  the  function;  2)  performance 
parameters;  3)  design  constraints;  and  4) 
requirements  for  reliability,  human 
performance,  safety,  operability, 
maintainability,  and  transportability. 
Following  synthesis  of  candidate  configuration 
items,  the  RAS  is  used  to  allocate  functional 


performance  requirements  to  individual 
system  elements  (hardware,  computer 
software,  personnel,  technical  manuals,  or 
facilities)  or  a  combination  of  elements. 
Although  the  RAS  is  a  useful  tool  throughout 
the  acquisition  cycle,  its  role  gradually 
diminishes  following  completion  of 
preliminary  design. 

The  physical  format  o*  a  particular 
application  of  the  RAS  is  very  flexible.  It 
may  be  expanded  either  vertically  or 
horizontally.  In  most  cases  it  will  have  no 
physical  form,  but  will  be  a  computer  file. 
Both  qualitative  and  quantitative  performance 
requirements  resulting  from  analysis  of  the 
function  are  identified  on  the  RAS.  These 
requirements  are  generally  identified  and 
grouped  by  engineering  specialty. 
Requirements  are  expanded  in  sufficient 
detail  to  provide  criteria  for  synthesizing  and 
evaluating  alternative  concepts  for  satisfying 
each  functional  requirement  in  terms  of 


Figure  6-8 

Requirements  Allocation  Sheet 
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combinations  of  system  elements.  The  RAS 
should  provide  all  design  requirements  and/or 
constraints  that  apply  to  the  system  element 
that  may  be  selected  or  designed  to  perform 
the  function.  Requirements  are  developed 
in  equal  depth  for  operational,  maintenance, 
test,  and  production  functions  identified  in 
FFBDs  within  any  tailoring  constraints 
established  in  the  SEMP. 

The  objectives  of  the  performance  and 
design  requirements  entries  on  the  RAS, 
when  used  to  support  the  system  level 
FFBDs  during  C/E  and  D/V,  are  to  establish 
functional  and  design  requirements  that  are 
included  in  the  design  sheet  (DS)  and 
subsequently  incorporated  into  the 
requirements  section  of  the  system  and 
development  specifications.  The  entries  also 
initiate  recognition  of  intrasystem  and 
intersystem  interface  and  facility 
requirements,  computer  software  and 
hardware  requirements,  and  personnel 
requirements. 

Performance  and  design  requirements 
entries  include  a  description  of  the  function, 
including  the  "why"  and  "what"  of  the 
function.  That  is,  the  descriptions  answer  the 
questions:  1)  Why  is  the  function  necessary? 

2)  Why  should  the  function  be  accomplished 
at  this  point  in  the  sequence  of  activities? 

3)  What  are  the  detailed  engineering  and 
support  characteristics  of  this  function? 

Performance  and  design  requirements 
entries  on  the  RAS  also  include  specific 
design  characteristics  created  by  the  function; 
in  other  words,  input,  output,  performance 
values,  and  allowable  quantitative  tolerances. 
They  also  include  applicable  maintenance 
constraints  such  as  check  out  limit,  calibration 


limitations  and  requirements,  and  accessibility 
requirements.  Detail  supplied  on  the  RAS 
should  be  sufficient  for  direct  use  as  design 
trade-off  criteria  which  initiate  and  control 
the  system  and  system  element  design. 
Sufficient  technical  detail  should  be  given  to 
allow  portions  of  one  or  more  RASs  to  be 
extracted  and,  in  conjunction  with  schematics, 
assembled  in  the  DS  as  integrated  design 
requirements. 

Personnel  task  analysis  and  performance 
requirements  for  all  tasks  are  listed 
separately  from  the  identified  functions. 
These  requirements  include  performance 
time  as  well  as  attributes  such  as  crew 
coordination,  job  knowledge,  safety,  skill,  and 
life  support.  Requirements  for  training  and 
training  equipment  are  also  identified. 
Where  multiple  personnel  are  involved,  TLSs 
may  also  be  required.  At  lower  levels  of 
analysis,  the  RAS  will  encompass  a  task 
analysis  which  provides  a  system  oriented 
basis  "or  the  development  of  technical 
manual  procedures  as  well  as  human 
engineering  analysis  and  other  task  analysis 
methodologies.  Time  constraints  either 
crewed  by  or  affecting  the  function  are 
identified.  Such  constraints  might  include 
computation,  countdown,  or  availability  times. 
All  technical  and  engineering  specialty 
requirements  which  constrain  or  have 
significant  influence  on  design  are  specified 
on  the  RAS.  These  requirements  include 
factors  such  as  power,  physical  dimension  and 
weight,  controlled  and  natural  environment, 
reliability,  maintainability,  and  human 
performance  capabilities  and  limitations. 

Functional  and  technical  interface 
requirements  are  separately  specified  and 
quantified  on  the  RAS.  Where  intersystem 
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interface  is  specified,  the  configuration  of 
that  system  is  specified,  together  with  the 
technical  characteristics  of  the  interface. 
When  any  of  the  above  entries  are  products 
of  trade  study  reports  (TSR),  other  back  up 
studies,  specifications,  or  other  sources,  the 
applicable  source  is  specifically  referenced. 

Facility  requirements  imposed  by  the 
performance  and  design  requirements  are 
identified  on  the  RAS.  Controlled  and 
natural  environmental  requirements  such  as 
temperature  and  humidity  ranges, 
illumination  and  noise  levels,  wind  and  snow 
loading,  precipitation,  penetration  and 
abrasion  effect,  and  atmospheric  pressure  are 
also  identified.  In  addition,  facilities  that 
must  be  developed  or  scheduled  on  a  long 
lead  basis  to  test  the  system’s  capability  to 
withstand  specific  environmental  and  utility 
requirements  such  as  power  (e.g.,  electrical 
and  hydraulic),  air  conditioning,  ventilation, 
and  heating  are  identified  on  the  RAS. 
These  requirements  begin  to  emerge  in  the 
C/E  phase  and  should  be  substantially 
complete  by  CDR. 


Functions  which  produce  complicated  or 
hazardous  requirements  involving  personnel 
will  generally  dictate  the  need  for  procedural 
information.  The  RAS  provides  the  means 
for  ensuring  that  the  developer  has 
programmed  development  of  the  procedural 
data. 

6.4.2  Performing  Technical  Allocation 

Performance  requirements  can  be  divided 
into  allocable  and  non-allocable  parameters. 
An  example  of  the  former  is  weight,  which 
is  progressively  divided  at  successively  lower 
levels.  An  example  of  the  latter  is  material 
and  process  standards,  which  are  applied 
directly  to  all  elements. 

Allocable  parameters  can  be  divided  into 
those  that  are  allocated  directly  and  those 
that  are  allocated  indirectly.  A  fire  control 
system  pointing  error  is  representative  of 
directly  allocated  requirements  in  which  the 
total  pointing  error  is  apportioned  first  to 
the  various  elements  and  then  to  subsystems 
and  their  components.  Indirectly  allocated 


Figure  6-9 

Example  of  Technical  Requirements  Allocation 


6-13 


requirements  are  those  that  require  an 
analysis  to  establish  performance  measures. 
An  example  of  this  would  be  the  conversion 
of  the  mission  requirement  for  aircraft  target 
detection  size  and  range  into  radiated  power, 
pulse  width,  and  timing  stability  which  could 
then  be  used  by  the  designer  of  the  radar 
system  in  sizing  his  hardware.  The  flowdown 
and  technical  allocation  process  is  illustrated 
in  Figure  6-9.  The  top  level  performance 
measures  are  used  to  derive  lower  level 
subsystem  requirements  for  configuring 
components.  The  process  is  documented  in 
RASs  which  define  each  requirement,  identify 
its  source,  and  show  the  allocation  to  the 
next  lower  level. 

It  is  important  to  note  that  as  a  result  of 
the  system  analysis  and  flowdown,  top  level 
functional  requirements  usually  become  lower 
level  performance  requirements.  For 
example: 

a.  System  -  Transmit  collected  data  in 
real  time  to  remote  ground  site 

b.  Segment  -  Provide  wideband  data  link 
from  spacecraft  to  relay 

c.  Element  -  Provide  10  MHz  link  at  17.0 
GHz 

d.  Subsystem  -  Provide  10  MHz  link  at 
17.0  GHz  with  10  W  effective  radiated  power 
for  20  minutes  maximum  per  orbital 
revolution. 

In  addition,  support  requirements  for 
power,  commands,  and  telemetry  are 
developed  and  quantified.  The  most  straight 
forward  application  of  allocation  is  the  direct 
apportioning  of  a  value  to  its  contributors. 


The  resulting  allocation  for  a  specific  area, 
such  as  pointing  error,  is  usually  referred  to 
as  a  budget.  The  technical  budget  represents 
an  apportionment  of  a  performance 
parameter  to  several  sources.  This  may  be 
a  top  down  allocation,  such  as  a  pointing 
error  budget,  or  a  bottom  up  summation, 
such  as  an  electrical  power  budget.  A 
discussion  of  two  critical  area  technical 
budgets  will  illustrate  the  nature  of  the 
process:  1)  pointing  error  and  2)  electrical 
power.  Characteristics  such  as  pointing  error 
or  electrical  power  distribution  would 
normally  become  parameters  for  technical 
performance  measurements  (TPM). 

6.4.2.1  Pointing  Error 

Allowable  pointing  error  is  a  critical  issue 
on  all  missile  and  spacecraft  programs. 
Typical  errors  range  from  several  tenths  of 
a  degree  to  a  few  arc  seconds  for 
astronomical  observatory  spacecraft.  In 
defining  the  error  budget,  it  is  necessary  to 
first  establish  those  hardware  and  software 
characteristics  that  contribute  to  the  error, 
otherwise  known  as  error  sources.  Sample 
error  sources  for  a  communication  spacecraft 
are  indicated  in  Figure  6-10.  Individual 
values  for  errors  would  be  obtained  from 
specifications  for  candidate  components, 
experience  from  similar  projects,  or 
extrapolation  of  experimental  data.  Where 
data  are  totally  lacking,  values  for  errors 
could  be  obtained  through  analysis. 
Typically,  a  minus  two  sigma  (0.95 
probability)  value  is  stated  in  the 
specification.  This  assumes  normal 
distribution  with  a  95  percent  confidence  in 
the  error  being  less  than  stated.  For  the 
above  example,  the  error  sources  are 
root-sum  squared  to  arrive  at  a  total,  since 
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Figure  6-10 

saaple  Error  Sources  for  Spacecraft 
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they  are  random  and  uncorrelated.  The 
allocated  pointing  requirements  would  be 
placed  in  subsystem  and  component 
specifications  as  appropriate. 

6.4.2J2  Electrical  Power 

Electrical  power  is  a  support  requirement 
determined  by  summing  the  individual 
component  loads.  It  is  usually  defined  by 
average  load,  peak  load,  and  a  profile  of 
power  demands  over  the  total  mission 
sequence.  In  developing  this  profile,  all 
electrical  items  in  the  design  must  be 
identified  and  a  mission  operational  scenario 
developed  to  define  equipment  operation 
and  duration.  Total  power  requirements  in 


each  mode  are  established  and  a  power 
profile  is  developed.  The  peak  and  average 
power  requirements  are  then  defined  to  size 
the  power  subsystem.  Because  some  items 
may  be  based  on  only  a  conceptual  design, 
and  because  power  needs  tend  to  increase, 
a  power  control  plan  is  often  used  that 
incorporates  margins  early  in  the  design 
process  to  allow  for  contingencies  that  may 
arise.  The  plan  also  provides  for  periodic 
review  of  requirements. 

6J  DOCUMENTATION 

Documentation  of  requirements  allocation 
is  an  essential  element  of  the  traceability 
process.  The  basic  allocation  document  is 
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the  RAS,  shown  previously  in  Figure  6-6. 
All  analyses  resulting  in  allocations  should  be 
documented  by  the  RAS  and  DS.  The 
concept  of  minimum  documentation  (tailoring 
of  documentation)  should  prevail.  In  most 
cases  a  minimum  amount  of  systems 
engineering  documentation  is  required  to  be 
formally  submitted  to  the  government. 

Technical  budgets  are  also  frequently 
maintained  in  a  performance  budget 
document.  This  provides  a  single  repository 
for  items  that  are  normally  dispersed  in  a 
number  of  specifications  and  permits  a  ready 
assessment  of  the  impact  of  a  change  in  any 
value.  Where  critical  budgets  are  involved, 
these  may  be  identified  as  TPM  parameters 
and  their  status  reported.  The  relation  of 
technical  budgeting  required  by  specialty 
specifications  (e.g.,  reliability  and  weight 


control)  to  systems  engineering  must  be  fully 
described  in  the  SEMP. 

6.6  SUMMARY 

Functional  analysis  forms  the  foundation 
for  all  systems  engineering.  The  principal 
ways  in  which  the  functional  analysis  is 
documented  are  FFBDs  and  RASs.  In  some 
cases,  TLSs  are  necessary  to  document  time 
critical  functions.  Figure  6-11  illustrates  some 
principal  relationships  between  elements  of 
the  systems  engineering  documentation.  In 
most  applications,  the  "sheets"  listed  will 
probably  be  the  computer  files  with  related 
subsets  of  data  elements.  Figure  6-11  also 
illustrates  the  relationships  among  primary 
hardware  documentation.  System  software 
data  are  discussed  in  greater  detail  in 
Chapter  20  of  this  guide. 


Figure  6-11 

Principal  Relationships  of  Primary 
Systems  Engineering  Documentation 
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CHAPTER  7 


SYSTEM  SYNTHESIS 


7.1  INTRODUCTION 

The  task  of  synthesis  is  defined  in 
MIL-STD-499A  [1]: 

"The  perfomumce,  configuration,  and 
arrangement  of  a  chosen  system  and  its 
elements  and  the  technique  for  their  test, 
support,  and  operation  shall  be  portrayed  in  a 
suitable  form  such  as  a  set  of  schematic  block 
diagrams,  physical  and  mathematical  models, 
computer  simulations,  layouts,  detailed 
drawings,  and  similar  engineering  graphics. 
These  portrayals  shall  illustrate  intra-\  and 
inter-system  and  item  interfaces,  permit 
traceability  between  the  elements  at  various 
levels  of  system  detail,  and  provide  means  for 
complete  and  comprehensive  change  control 
This  portrayal  shall  be  the  basic  source  of  data 
for  developing,  updating,  and  completing  (a) 
the  system,  configuration  item  (Cl),  and  critical 
item  specifications;  (b)  interface  control 
documentation;  (c)  consolidated  facility 
requirements;  (d)  content  of  procedural 
handbooks,  placards,  and  similar  forms  of 
instructional  data;  (e)  task  loading  of 
personnel;  (f)  operational  computer  programs; 
(g)  specification  trees;  and  (h)  dependent 
elements  of  work  breakdown  structures  (WBS)”. 

One  of  the  main  objectives  of  the  systems 
engineering  process  is  to  strike  a  balance 
among  functional  performance  requirements, 
system  constraints,  and  system  effectiveness 


criteria  in  determining  the  appropriate  design 
concept(s).  Synthesis,  or  conceptual  design, 
is  the  activity  which  assures  that  system 
influences  are  given  the  proper  consideration 
in  arriving  at  a  design  concept.  It  is  the  point 
in  the  systems  engineering  process  where 
engineering  creativity  and  technology  are 
brought  to  bear  in  the  creation  of  a  system 
or  design  concept  which  best  meets  the 
stated  system  requirements.  Synthesis 
considers  the  results  of  various  technical  and 
design  studies  as  well  as  the  requirements 
delineated  from  the  functional  analyst  effort. 
It  requires  the  inputs  from  all  of  the 
technology  and  engineering  specialty  areas 
that  have  a  bearing  on  the  system  or  design 
concept  and  should  take  into  account  the 
latest  technological  advances  in  the  areas  of 
design,  producibility,  and  supportability. 

Synthesis  is  performed  initially  to 
postulate  possible  technical  approaches  using 
the  results  from  the  functional  analysis 
activity.  In  supporting  each  technical 
approach,  one  or  more  system  concepts 
(arrangements  of  system  elements  which  will 
satisfy  the  functional  performance 
requirements)  will  be  identified.  Later,  during 
successive  iterations  of  the  systems 
engineering  process,  one  or  more  design 
concepts  will  be  synthesized  for  each  system 
concept.  The  configuration  and  arrangement 
of  system  elements  may  be  portrayed  in  any 
suitable  form;  however,  the  two  most  widely 
accepted  synthesis  documentation  tools  are: 
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1)  the  schematic  block  diagram  (SBD)  and 

2)  the  concept  design  sheet  (CDS). 

Synthesis  of  solutions,  developed 
during  the  Concept  Exploration/  Definition 
(C/E)  phase,  is  accomplished  only  to  the 
level  to  which  the  government  wishes  to 
constrain  the  competing  Concept 
Demonstration/  Validation  (D/V)  phase 
contractors.  During  the  Full  Scale 
Development  (FSD)  phase,  synthesis 
develops  still  greater  levels  of  detail  until 
drawings  and  specifications  are  produced 
which  can  be  used  in  the  fabrication  and 
assembly  of  hardware,  and  the  coding  and 
assembly  of  software  programs.  Portrayal  of 
a  synthesized  system  in  terms  of  its  elements 
(equipment,  software,  facilities,  personnel, 
and  procedural  data)  will  provide  a  source  of 
data  for  equipment  design  documentation, 
interface  control  documentation,  consolidated 
facility  requirements,  handbooks  and  guide 
books,  task  loading  of  personnel,  specification 
tree,  and  work  breakdown  structures  (WBS). 

In  summary,  synthesis  is  the 
development  of  "how"  answers  to  the  "what" 
tasks  in  the  functional  description  of  the 
proposed  system. 

12  SYNTHESIS  TOOLS 

System  synthesis  originates  from  the 
definition  of  system  heirarchy  and  the 
specification  tree.  The  government  often 
defines  the  functional  requirements  of  the 
system,  as  well  as  constraints  on  the  system, 
such  as  the  use  of  existing  equipment.  These 
are  used  to  define  the  overall  system  level 
functional  flow  depicted  in  the  functional 
flow  block  diagram  (FFBD).  The  CDS  is 


used  to  collect  the  performance  requirements 
and  constraints  as  delineated  by  functional 
analysis  that  apply  to  an  individual  subsystem 
or  end  item,  and  to  describe  a  design 
approach  for  meeting  the  requirements. 
When  sufficient  functional  identification  and 
decomposition  has  occurred,  the  configuration 
and  arrangement  of  system  elements  are 
portrayed  in  SBDs.  These  detailed  block 
diagrams  are  the  basis  for  system  synthesis  as 
shown  in  Figure  7-1,  by  serving  as  the  basis 
for  models  of  the  system.  These  models  can 
either  be  physical  or  mathemetical.  The 
development  of  models  requires  the  systems 
engineer  to  organize,  evaluate,  and  examine 
the  validity  of  his/  (her)  thought  processes. 
Their  use  permits  an  optimization  of 
hardware  and  software  parameters,  allows 
perfromance  predictions  to  be  made,  permits 
operational  sequences  to  be  derived,  and 
allows  evaluation  to  the  optimun  allocation 
of  functional  requirements  between  the 
system  elements. 

7.2.1  Schematic  Block  Diagrams 

A  schematic  block  diagram  (SBD) 
is  one  of  the  primary  tools  for  the  system 
synthesis  by  serving  as  the  basis  for  models 
of  the  system.  They  are  developed  at 
successively  lower  levels  as  analysis  proceeds 
to  define  lower  level  functions  within  higher 
level  requirements.  These  requirements  are 
further  subdivided  and  allocated  using  the 
Requirements  Allocation  Sheet  (RAS). 
SBDs  provide  visability  of  related  system 
elements,  and  traceability  to  the  RAS  and 
other  systems  engineering  documentation. 
Furthermore,  SBDs  1)  depict  a  complete 
response  to  the  functional  need  which  meets 
the  initial  input  requirments,  2)  depict 
compatibility  between  the  elements  of  the 
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Figure  7-1 

Schematic  Block  Diagram 
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system  and  interfacing  systems/  subsystems, 
3)  permit  traceability  between  elements  and 
their  functional  origin,  and  4)  ensure 
complete  and  comprehensive  change  control. 

The  SBD  shows  selected  functions  and 
data  interfaces  within  the  system.  A 
simplified  SBD,  Figure  7-1,  shows  the 
components  which  may  comprise  a 
subsystem,  and  the  data  which  may  flow 
between  them.  An  expanded  version  is 
usually  developed  which  displays  the  detailed 
functions  performed  within  each  component 


and  their  interrelationships.  For  complex 
systems,  this  may  then  be  developed  into  a 
logic  diagram  for  auditing  the  schematics 
produced  by  design  engineers.  This  audit  is 
a  critical  systems  engineering  function.  The 
SBD  is  used  to  develop  Interface  Control 
Documents  (ICDs)  and  provide  an  overall 
understanding  of  system  operations.  A  key 
goal  of  the  SBD  is  to  define  "modular  units." 
Modular  units  can  be  characterized  as  follows 
[2]: 

a.  Implementing  a  single,  independent 
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function 

b.  Performing  a  single  logical  task 

c.  Having  a  single  entry  and  exit  point 

d.  Being  separately  testable. 

Desirable  attributes  of  the  modular  units 
include  low  coupling,  high  cohesion,  and  low 
connectivity.  Coupling  between  modules  is  a 
measure  of  their  interdependence,  or  the 
amount  of  information  shared  between  two 
modules.  Decoupling  of  modules  eases 
development  risks  and  makes  later 
modifications  easier  to  implement.  Cohesion 
(also  called  binding)  is  the  similarity  of  tasks 
performed  within  the  module.  Various  levels 
of  cohesion  have  been  defined  as  follows 
(highest  to  lowest)  [3]: 

a.  Functional-  all  functions  contribute 
directly  to  performing  a  single  function 

b.  Sequential-  data  flow  is  processed 
sequentially  from  input  to  output 

c.  Communicational-  operates  on  common 
data 

d.  Procedural-  follows  a  logic  flow  path 

e.  Temporal-  performs  differing  functions 
at  the  same  time 

f.  Logical-  performs  several  similar  but 
unrelated  functions. 

Connectivity  is  the  reference  from  internal 
elements  within  one  module  to  internal 
elements  within  another  module.  High 
connectivity  is  undesirable  in  that  it  creates 


complex  interfaces  which  may  impede  design, 
development,  and  testing. 

7.2.2  Physical  Modeling 

Physical  models  can  be  either  full  size, 
scale,  hardware,  or  analog  representations  of 
the  system.  Where  human  interaction  is 
involved,  such  as  manned  vehicles  or  control 
consoles,  the  models  are  frequently  built  full 
size,  in  either  soft  (foam  core  or  plywood)  or 
hard  (metal)  construction.  Human 
engineering  personnel  use  them  to  verify 
operator  capabilities,  optimum  location  of 
controls,  and  response  times,  as  well  as  to 
establish  maintainability  characteristics  and 
ensure  maximum  efficiency  of  operation. 
Full  scale  models  are  also  used  by  designers 
to  provide  a  three  dimensional  representation 
of  complex  structures  to  facilitate  design  of 
features  such  as  cable  harness  routing,  box 
placement,  and  access  opening  location. 

For  missiles  and  aircraft,  scale  models  are 
tested  in  wind  tunnels  to  establish  lift  and 
drag  characteristics  for  subsequent  use  in 
computer  flight  simulations.  Scale  ship  and 
submarine  hull  models  are  tested  in  model 
basins  (towing  tanks)  to  establish  hull 
behavior  and  expected  performance  when 
subjected  to  a  range  of  operating  conditions 
(such  as  wave  forms  or  sea  states).  Scale 
models  also  assist  designers  in  visualizing  a 
concept  or  operation  where  a  full  scale 
model  is  either  impractical  or  too  costly,  such 
as  a  space  station. 

Engineering  hardware  models  are  often 
used  to  provide  proof  of  functional  operation 
or  to  establish  critical  performance 
characteristics.  A  typical  example  would  be 
a  breadboard  or  brassboard  (for  radio 
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frequency  components),  which  may  bear  little 
resemblance  to  the  final  operational 
configuration,  but  permits  demonstration  of 
basic  principles.  Data  gathered  provide  early 
verification  and  permit  optimization  of  the 
final  design. 

Analog  models  are  a  physical 
representation  of  a  system  in  an  alternate 
form.  Typicallv,  electrical  circuitry  is  used  to 
represent  a  mechanical  system.  A  common 
use  is  in  control  system  analysis.  The  analog 
model  is  essentially  a  computer  simulation 
using  amplifiers,  capacitors,  and  feedback  to 
solve  mathematical  equations  representing 
physical  system  functions.  The  nature  of  the 
circuitry  employed  permits  real  time 
parameter  variations  which  greatly  reduce 
analysis  time.  Limitations  in  accuracy  and 
complexity,  however,  restrict  the  use  of 
analog  models,  especially  as  digital  computers 
have  grown  in  capability.  Their  current  use 
is  primarily  in  hybrid  operation  with  digital 
computers  representing  the  control  elements 
in  large  system  simulations. 

1.23  Mathematical  Modeling 

A  mathematical  model  is  an  abstract 
representation  (without  regard  to  physical 
implementation)  of  a  system.  For  this 
reason,  it  provides  a  means  of  developing 
quantitative  performance  requirements  from 
which  candidate  designs  can  be  developed. 
Static  models  are  those  that  depict  conditions 
of  state,  such  as  the  loading  of  a  mechanical 
structure.  If  the  equilibrium  condition  is 
changed  by  altering  the  loading  conditions, 
new  values  for  the  load  paths  may  be 
obtained  analytically,  but  the  model  does  not 
indicate  the  manner  in  which  the  load  paths 
achieved  their  new  state.  Should  it  be 


desired  to  optimize  the  load  paths  according 
to  the  capability  of  structural  members,  a 
numerical  solution  would  be  required. 

Dynamic  models  depict  conditions  that 
vary  with  time,  such  as  the  action  of  an 
autopilot  in  controlling  an  aircraft.  Simple 
dynamic  models  can  be  solved  analytically, 
and  the  results  represented  graphically. 
However,  simple  models  do  not  usually  exist 
in  large  systems  and,  therefore,  numerical 
methods  are  used. 

Numerical  methods  involve  the  use  of 
digital  computer  simulations.  While  this  can 
be  an  expensive  undertaking,  it  provides 
advantages  of  timeliness,  versatility,  and  ease 
of  parameter  variability,  and  is  usually  less 
costly  than  building  and  testing  an  actual 
system.  Development  of  a  computer 
simulation  is  performed  in  the  steps  discussed 
below  [23: 

a.  PiQblem  Definition  -  The  objectives  and 
scope  of  the  study  must  be  clearly  defined  in 
unambiguous  terms.  Any  limitations  should 
be  identified.  A  specification  should  be 
prepared  that  defines  the  functions  to  be 
performed  by  the  simulation,  and  all 
performance  requirements  which  it  must 
satisfy  to  meet  study  objectives.  Each 
requirement  must  be  testable  after  the 
simulation  is  developed. 

b.  Math  Model  Formulation  -  The 
equations  which  define  functional 
performance  are  then  developed  and  a  flow 
chart  is  prepared  which  shows  all  processes, 
data  paths,  decision  points,  files,  inputs,  and 
outputs. 

c.  Program  Construction  -  Software 
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modules  are  gathered  or  new  programs  are 
coded,  assembled,  and  checked  out  to  ensure 
satisfactory  integration. 

d.  Verification  -  A  test  program  is  planned 
and  conducted  which  demonstrates  that  the 
simulation  meets  the  requirements  of  its 
specification. 

e.  Experimental  Design  -  Appropriate 
scenarios  and  input  conditions  are  developed 
for  the  case  under  study.  These  should 
include  end  or  limit  conditions  for  which  the 
outcome  is  known. 

f.  Validation  -  Validation  is  different  from 
verification  in  that  it  demonstrates  that  the 
model  portrays  the  actual  situation  and 
interacts  properly  with  the  real  world.  The 
situations  chosen  for  validation  must  have  a 
known  outcome. 

Once  validated,  the  simulation  may  then 
be  employed  to  establish  and  optimize 
parameters  and  to  verify  the  selected  design. 
Development  of  simulations  is  an  iterative 
process,  with  changes  in  requirements  and 
programs  being  incorporated  as  experience 
is  gained.  Simulations  are  classified  as  either 
continuous  or  discrete.  In  continuous 
programs,  independent  variables  are  a 
function  of  time.  This  is  useful  in 
establishing  the  behavior  of  the  system  during 
transient  responses  to  perturbations. 
Discrete  models  change  state  only  in  steps; 
interstate  transients  are  not  considered. 

A  discrete  model  often  results  from 
disruption  of  the  system  status  caused  by  the 
allocation  and  reallocation  of  resources  within 
the  system.  Queueing  is  an  important 
consideration  in  systems  that  can  be 


described  by  discrete  event  models.  Discrete 
models  can  be  further  subdivided  into 
process  and  event  oriented  models.  A 
process  oriented  model  views  the  world  as  a 
set  of  fixed  facilities  which  are  used  to 
service  active  transactions  that  are  created 
and  moved  through  the  system.  An  event 
oriented  model  views  the  world  as  a  series  of 
events  which  occur  at  scheduled  times  and 
searches  for  facilities  to  process  them. 

73  DRAWINGS  AND  LISTS 

Systems  engineering  prepares  and  audits 
system  level  design  documentation  which  is 
piaced  under  configuration  management.  The 
initial  definition  of  components,  provided  by 
systems  engineering,  is  a  listing  which 
identifies  a1'  ^mponents  in  the  system,  their 
numbers,  and  intended  use.  This  is  used  by 
the  design  engineering  organization  as  the 
basis  for  equipment  and  weight  lists. 
Eventually,  these  will  become  a  parts  list  to 
identify  and  track  hardware  and  computer 
software  items. 

In  the  case  of  a  spacecraft,  the  primary 
system  level  data  developed  by  the  design 
engineering  organization  may  be  the  vehicle 
inboard  profile,  envelope  drawings,  and 
facility  layout  drawings.  The  inboard  profile 
drawing  will  establish  the  location  of  all 
comp»onents  on  the  vehicle  and  assure  that 
adequate  space  is  available.  The  envelope 
drawing  will  show  the  coordinates,  overall 
physical  dimensions,  view  angles,  and 
equipment  attachment  points  and  serve  as 
the  basis  for  interface  requirements.  The 
facility  layout  drawing  will  determine  facility 
space  requirements  for  equipment  and  ensure 
that  adequate  area  is  available  for  personnel 
operations/  maintenance. 
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CHAPTER  8 


EVALUATION  AND  DECISION: 
TRADE  STUDIES 


8.1  INTRODUCTION 

Trade  studies  are  performed  throughout 
development  as  an  essential  part  of  the 
systems  engineering  process  outlined  in 
Chapter  5  of  this  guide.  Trade  studies  are 
controlled  by  systems  engineering  to 
integrate  and  balance  all  design-for  and 
engineering  specialty  requirements. 

As  a  formal  decision  analysis  method, 
trade  studies  are  used  to  solve  any  complex 
problem  where  there  is  more  than  one 
selection  criterion,  and  provide  documented 
decision  rationale  for  review  by  a  higher 
authority.  These  analyses  are  equally 


necessary  for  establishing  system 
configurations  and  for  accomplishing  detailed 
design  of  individual  components.  The  trade 
study  method  is  equally  applicable  to 
budgeting,  source  selection,  test  planning, 
logistics  development,  production  control, 
and  design  synthesis. 

The  role  of  trade  studies  evolves  with  the 
acquisition  process,  as  shown  in  Figure  8-1. 
During  the  Concept  Exploration/  Definition 
(C/E)  and  the  Concept  Demonstration/ 
Validation  (D/V)  phases,  trade  studies  are 
used  to  establish  the  system  configuration. 
During  Full  Scale  Development  (FSD), 
trade  studies  are  employed  to  assist  in 


Figure  8-1 

Trade-Off  Analysis  in  the  Acquisition  Process 
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selecting  component/part  designs.  Later,  as 
the  system  enters  the  Production  phase,  trade 
studies  support  make-or-buy,  process,  rate, 
and  location  decisions  as  well  as  examination 
of  all  proposed  design  changes.  Control  ot 
systems  engineering  trade  studies  throughout 
the  acquisition  cycle  (to  balance 
considerations  such  as  producibility, 
testability,  survivability,  compatibility,  stability, 
supportability,  and  reliability  with  cost, 
schedule,  and  performance  objectives)  is  the 
primary  means  of  executing  systems 
engineering  responsibilities.  Some 

applications  of  the  trade  study  method  in  the 
systems  engineering  process  are  indicated  in 
Figure  8-2. 


8.2  BASIC  METHODOLOGY 

The  trade-off  analysis  methodology 
provides  a  structured,  analytical  framework 
for  evaluating  a  set  of  alternative  concepts  or 
designs.  Figure  8-3  shows  the  basic  steps  of 
the  analysis,  which  are  outlined  in  the 
following  paragraphs. 

8.2.1  Define  Objectives  and  Requirements 

Analysis  objectives  and  requirements  must 
be  expressed  in  precise,  explicit  terms  to 
serve  as  the  basis  for  sound  decisions.  They 
should  define  the  need,  the  user,  and  the 
availability  of  resources  bounding  the  scope 


Figure  8-2 

Trade-Off  Analysis  in  the  Systems  Engineering  Process 
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Figure  8-3 

Trade-Off  Analysis  Methodology 


of  the  analysis.  The  source  for  these 
objectives  will  be  systems  engineering 
documentation  such  as  the  Functional  Flow 
Block  Diagram  and  Requirements  Allocation 
Sheet  (FFBD  and  RAS).  This  will  provide 
a  firm  foundation  for  identifying  the  range  of 
alternatives  and  the  decision  criteria. 

Examples  of  partial  objectives/ 
requirements  for  a  trade-off  study  involving 
design  of  the  aft  crane  configuration  on  a 
logistic  ship  for  the  Navy  (see  Reference  [3]) 
are: 

a.  Loading  must  be  accomplished  in  less 
than  48  hours. 

b.  The  ship  must  have  the  capability  to 
carry  out  roll-on/roll-off  (RO/RO)  operations 
in-stream  as  well  as  pierside. 

c.  Discharge  time  at  pierside  must  not 
exceed  24  hours. 

8.2.2  Identify  Alternatives 

Alternatives  for  consideration  will  be 
either  predetermined  (in  the  case  of  a  design 
competition,  they  will  be  the  various 
proposed  designs)  or  developed  specifically 
for  the  analysis.  Candidate  alternatives  may 
be  the  product  of  systems  engineering 
synthesis  activities  and  represent  existing 
(standard),  modified,  or  original  designs. 
Candidate  alternatives  should  reflect  the 
widest  possible  range  of  distinctly  different 
solutions  if  the  overall  goal  of  optimized 
system  design  is  to  be  achieved. 

Next,  candidate  alternatives  identified 
through  unconstrained  synthesis  or 
brainstorming  may  be  screened  based  on 


their  ability  to  solve  the  problem.  This 
ensures  that  the  analysis  effort  does  not 
waste  time  on  nonproductive  solutions.  A 
second  screening  may  be  performed  on  the 
basis  of  attainability/affordability:  Are  the 
candidate  alternative  solutions  achievable 
within  time  and  budgetary  constraints? 

Remaining  candidate  alternatves  become 
the  decision  alternatives.  These  alternatives 
are  described  fully  and  carefully.  Sufficient 
detail  must  be  available  to  judge  the  relative 
worth  of  each  workable,  attainable 
alternative.  If  an  insufficient  number  of 
candidate  alternatves  survives  the  screening 
process,  the  study  constraints  should  be 
reexamined  and  all  candidate  alternatives 
rescreened,  or  the  synthesis  and  possibly 
functional  analysis  activity  must  be  reinitiated. 
In  the  example  given  in  Paragraph  8.2.1, 
three  configuration  alternatives  were 
considered: 

a.  Configuration  1  -  Two  70-ton  gantry 
cranes  at  Location  1 

b.  Configuration  2  -  Two  50-ton  revolving 
boom  cranes  at  Location  2 

c.  Configuration  3  -  Two  50-ton  revolving 
boom  cranes  at  Location  3. 

8.23  Formulate  Selection  Criteria 

Selection  criteria  are  standards  for  judging 
achievement  of  required  operational 
effectiveness/suitability  characteristics,  or 
resolution  of  technical  or  operational  issues. 
The  criteria  may  include  quantitative  goals 
(desired  value  of  the  attribute),  where 
possible,  and  thresholds  beyond  which  the 
characteristic  is  unsatisfactory  (specification 
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limits).  Good  selection  criteria  must:  Selection  criteria  may  be  drawn  from 

systems  engineering  documentation  based  on 

a.  Differentiate  meaningfully  between  program  requirements;  military  and 

alternatives  without  bias  department  guidance  and  standards;  and 

design-for  and  specialty  requirements.  These 

b.  Relate  directly  to  purpose  of  the  sources  vary  in  importance  based  on  the 

trade-off  analysis,  including  established  stage  of  program  development  and  design 

requirements  and  high-interest  concerns  maturity.  Typical  sources  for  selection 

criteria  at  each  phase  of  the  acquisition 

c.  Be  stated  as  broadly  as  possible  process  are  shown  in  Figure  8-4.  These  or 

similar  program  documents  typically  provide 

d.  Be  able  to  be  measured  or  estimated  at  performance,  schedule  and  cost  ranges/ 

reasonable  cost  thresholds,  and  background/  decision 

information.  Regardless  of  the  sources  used 

e.  Be  independent  of  each  other  at  all  and  the  advice  obtained,  final  selection  must 

levels  be  made  by  the  decision  maker.  The  value 

of  the  trade-off  analysis  effort  is  proportional 

f.  Be  universally  understood  by  evaiuators.  to  the  decision  maker’s  ability  and  willingness 


Figure  8-4 

Sources  of  Design  Trade  Study  Decision  Criteria 
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to  include  all  objective  and  subjective 
decision  criteria.  Regular,  efficient  guidance 
on  appropriate  decision  criteria  is  one  of  the 
primary  products  of  the  systems  engineering 
organization. 

8.2.4  Weight  the  Criteria 

Selection  criteria  are  weighted  by  the 
decision  maker  according  to  their  relative 
importance  in  determining  the  effectiveness 
of  alternatives.  To  ensure  the  objectivity  of 
the  subsequent  analysis,  weighting  factors 
de\eloped  by  the  decision  maker  may  be 
withheld  from  the  analysts  who  do  the 
performance  evaluation. 

Weighting  follows  a  logical  breakdown 
such  as  the  one  illustrated  in  Figure  8-5  for 
a  ship  design  program.  Essentially,  the 
numerical  scale  used  is  coincidental,  provided 
that  it  is  consistently  distributed  down  the 


criteria  tree.  In  this  example,  criteria  are 
classified  by  their  relative  contribution  to 
mission  capability  including  speed/  endurance, 
logistics,  cargo  capacity,  safety,  and  cargo 
capability.  First,  effectiveness  measures  are 
examined  for  their  contribution  to  objectives 
for  the  system;  then  each  criterion  is 
weighted  according  to  its  perceived 
contribution  to  the  effectiveness  measures. 
The  extent  of  the  breakdown  required  is 
determined  by: 

a.  The  level  at  which  performance 
evaluation  is  possible 

b.  The  level  at  which  separate  performance 
specifications  have  been  established. 

Numerical  weights  are  given  to  reduce  the 
effect  of  evaluator  bias  on  the  analysis. 
Numerical  weighting  allows  the  decision 
maker  to  obtain  an  objective  assessment  of 


Figure  8-5 

Sanpie  Criteria  Weighting 
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the  alternatives.  In  addition,  numerical 
treatment  facilitates  comparison  among 
criteria  that  are  not  related.  For  instance,  in 
this  example,  cargo  capacity  is  twice  as 
important  as  speed  and  endurance,  and  only 
slightly  less  important  than  cargo  capability. 
The  advantages  in  relative  simplicity, 
efficiency,  and  objectivity  of  this  approach  far 
exceed  the  effort  required  to  assign 
numerical  weights.  Decision  makers  who 
claim  that  they  cannot  assign  numerical 
weights  to  the  criteria  should  realize  that 
decisions  are  based  on  quantified  criteria 
whether  that  quantification  is  subconscious 
(unsystematic  and  undocumentable)  or 
objective/  numerical  (systematic  and 
documentable). 

Where  a  program  maintains  an  overall 
system  effectiveness  model  and  has 
operational  data,  this  weighting  process  can 
be  very  objective.  In  cases  where  such  a 
foundation  is  not  available,  decision  support 
techniques  can  be  used  to  render  subjective 


evaluations  more  reliable.  In  the  ship 
example  used  in  this  chapter,  the  authors 
used  the  analytic  hierarchy  process  (see 
Reference  [3])  to  set  criteria  weights  using 
a  paired  comparison  technique.  Engineers 
and  managers  from  the  design  team  were 
asked  to  prepare  data  input  sheets  that 
compared  attributes  at  each  level  on  a 
one-to-one  basis.  A  typical  input  sheet  is 
shown  in  Figure  8-6.  Data  were  entered  into 
a  computer  for  analysis,  consolidation,  and 
normalization  into  matrix  form.  The 
mathematical  technique  of  eigenvector 
analysis  was  then  applied  to  the  normalized 
matrix  to  determine  the  relative  weightings 
of  all  components  at  each  level.  Once  the 
weighting  factors  of  all  elements  of  the 
hierarchy  were  derived,  the  "contributing 
weight"  of  any  one  attribute  could  be 
calculated  by  multiplying  the  weights  of  its 
associated  category  headings  by  its  weighting 
factor.  Data  from  respondents  were 
summarized  and,  with  minor  adjustments, 
resulted  in  the  priorities  shown  in  Figure  8-5. 


Figure  8-6 

Sample  Prioritisation  Input  Sheet 
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This  technique  of  pairwise  comparisons  has 
been  shown  to  give  mrso  repeatable 
weightings  than  direct  estimation  of  the 
relative  attribute  priorities. 

8.2.5  Prepare  Utility  Functions 

Although  not  necessary  for  every  trade 
study  application,  utility  curves  are  a  good 
technique  for  translating  diverse  criteria  to  a 
common  scale.  For  example,  in  the  trade 
study  illustrated,  how  does  one  compare 
speed  (in  knots)  with  endurance  (in  nautical 
miles),  or  cargo  capacity  (in  number  of 
vehicles)  with  cargo  capability  (in  hours  at 
dock)?  Utility  functions  provide  this 
mediating  capability.  Briefly,  utility  curves 
assume  that  changes  in  the  performance 
associated  with  a  particular  criterion  can  be 
translated  into  a  utility  score.  This  utility 
score  may  range  from  0  to  1,  with  the  lower 
bound  on  the  possible  value  of  an  attribute 
being  assigned  a  utility  of  0,  and  the  upper 
bound  being  assigned  a  utility  of  1.  The 
range  of  the  utility  curve  encompasses  the 
range  of  acceptable  or  realistic  alternatives. 


For  example,  if  it  were  determined  that  the 
baseline  design  would  allow  a  top  speed  to 
range  anywhere  from  27  to  33  knots,  27 
knots  would  be  assigned  a  utility  curve  value 
of  0  and  33  knots  would  be  given  a  utility 
curve  value  of  1.  Utility  curves  may  be 
developed  using  engineering  judgment  or  a 
more  quantitative  approach,  such  as 
assessment  of  the  probability  of  the  ship’s 
survival  during  a  mission  as  a  function  of  its 
speed.  Figure  8-7  illustrates  one  theoretical 
utility  curve  for  ship  speed.  The  utility  value 
of  an  attribute  is  multiplied  by  the 
contributing  weight  of  the  attribute  to 
determine  the  change  in  overall  mission 
capability  of  the  ship.  Figure  8-8  shows 
other  sample  utility  curves. 

Reference  [1]  makes  the  distinction 
between  three  approaches  to  establishing 
utility  scales:  1)  absolute  scaling,  2)  ratio 
scaling,  and  3)  relative  scaling.  The  examples 
used  in  this  chapter  all  infer  the  use  of 
absolute  scales.  This  approach  is  the  most 
desirable  for  military  system  evaluation.  It 
assumes  that  by  analysis  or  initiative,  it  is 


Figure  8-7 

Sample  Utility  Curve  for  Ship  Speed 
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Figure  8-8 

Sample  Utility  Curves 


possible  to  conceptualize  a  "perfect"  system  political  saleability)  where  only  subjective 

and  to  predict  a  level  of  performance  with  (high,  medium,  low)  evaluation  is  possible, 

respect  to  each  attribute  for  each  alternative  For  example,  if  safety  was  an  essential  trade 

being  evaluated.  study  criterion,  it  could  be  included  as  a 

relatively  scaled  attribute  using  a  paired 
In  cases  where  an  attribute  is  difficult  to  comparison  process  and  scaling  methodology 

quantify  or  measure,  the  evaluation  might  similar  to  that  suggested  in  Reference  [2]. 

establish  a  ratio  scale  (see  Reference  11])  or 

use  the  analytic  hierarchy  process  (see  Utility  curves  for  a  given  trade  study  must 

Reference  [2])  to  establish  a  relative  scale  use  consistent  scales  (e.g.,  between  0  and  1) 

for  utility  values.  The  ratio  and  relative  so  as  not  to  inadvertently  weight  the  scores, 

scaling  approaches  identify  the  "best"  These  models  also  must  assume  the 

alternative  through  a  structured  comparison  independence  of  criteria.  The  "zero  point" 

of  alternatives.  These  approaches  are  most  of  each  curve  indicates  the  level  of 

valuable  in  considering  nontechnical  performance  which  no  longer  provides  value 

parameters  (such  as  cost,  development  time,  to  system  performance  or  effectiveness.  The 


specification  values,  since  minimum 
acceptable  values  are  usually  the  cut-off 
beyond  which  alternatives  are  "not  worthwhile 
to  pursue,"  rather  than  "without  value." 

Graphic  utility  curves  are  not  necessary 
for  every  criterion.  Where  linear 
relationships  are  assumed  between  utility  and 
performance,  simple  tables  can  be 
established.  Tabular  scoring  plans  could 
replace  graphic  charts  for  any  criterion; 
however,  some  fixed  plan  for  scoring 
performance  evaluations  must  be  established 
before  the  evaluations  are  conducted. 

8.2.6  Evaluate  Alternatives 

After  utility  functions  have  been 
established,  the  performance  of  each 
alternative  is  estimated  with  respect  to  each 
criterion.  Performance  estimates  are 
produced  by  evaluators  from  testing,  vendor 


sources,  parametric  analysis,  simulation, 
experience,  comparison,  or  other  available, 
affordable,  and  dependable  methods. 

The  scoring  plan  represented  by  the 
utility  curves  is  then  used  to  convert  these 
performance  estimate  s  into  effectiveness 
measures  by  assigning  a  score  for  each 
performance  level.  For  example,  in  Figure 
8-7,  shown  previously,  an  alternative 
evaluated  as  having  an  expected  speed  of 
31.5  knots  would  receive  a  score  of  .50,  while 
an  expected  speed  of  33  knots  would  receive 
a  score  of  1.0. 

Scores  are  collected  and  summarized  in 
tabular  form.  A  table  is  developed  which 
shows  assigned  scores  on  the  same  numerical 
scale  for  all  criteria  and  all  alternatives.  The 
decision  maker  applies  criteria  weights  to  the 
evaluation  results  to  complete  the  effort. 
Figure  8-9  shows  a  sample  weighted  summary 


Figure  8-9 

Sample  Weighted  Summary  Table 
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Figure  8-10 

Aft  Crane  Study  Weighted  Summary  Table 
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0.38 

0.2736 

0.5 

036 

0  5 

0.36 

No.  of  vehicles 

0.21 

0.788 

0.1654 

0.788 

0.1654 

0.687 

0.1442 

0.71 

0.1491 

No.  of  containers 

0.07 

0.6 

0.042 

0.6 

0.042 

0 

0 

0 

0 

—Mas  tank  condition 

0.74 

0.4199 

0.3559 

0.3731 

0.37S7 

No.  of  helo# 

0.71 

0.5 

0.355 

0.5 

0.355 

0.6 

0.355 

0.5 

0.355 

No.  of  vehicles 

0.22 

0.162 

0.0356 

0.156 

0.0343 

0.08 

0.0176 

0.162 

0.0356 

No.  of  containers 

0,07 

0.856 

0.0599 

0.856 

0.0599 

0 

0 

0 

0 

—■Mas  helo  condition 

0.26 

0.1171 

0.1168 

0.0968 

0.1015 

Canto  capacity 

0.33 

0.1772 

0.1560 

0.1551 

0.1578 

— Speed:  mu  power 

0.29 

0.49 

0.1421 

0.49 

0.1421 

0.49 

0.1421 

0  49 

0.1421 

— Endurance 

0.71 

1 

0.71 

1 

0.71 

1 

0.71 

1 

0.71 

Speed  and  endurance 

0.16 

0  1363 

0.1363 

0.1363 

0.1363 

Logistics/K&M 

0.09 

0.5 

0.045 

0.4 

0.036 

0.6 

0.054 

0.6 

0.054 

Safety 

0  06 

05 

0.03 

0  5 

003 

0.5 

0.03 

0  5 

0.03 

Mission  capability 

Cost  (millions! 
Cost/capability  ratio 

0.5673 

60 

105.7.Vr 

0  5371 
65.72 
122.3509 

0.5542 

58.48 

105.5156 

0.5314 

58.58 

110.2165 

table.  The  weighted  summary  table  that 
was  used  in  the  study  in  Reference  [3]  is 
included  as  Figure  8-10.  It  shows  a  slightly 
different  format  from  Figure  8-9. 

8.2.7  Perform  Sensitivity  Check 

A  sensitivity  analysis  must  be  performed 
to  determine  the  value  of  results  to  the 
decision  maker.  Where  the  total  weighted 
scores  of  several  alternatives  are  proximate, 
a  small  change  in  the  estimated  performance 
of  any  alternative  against  any  criterion  may 
change  the  decision.  This  is  especially 
important  where  performance  estimates  are 
developed  without  benefit  of  operational 
data.  In  these  instances,  it  may  be  useful  to 
indicate  a  range  of  estimated  performance 
values  having  a  known  confidence  level  which 
can  be  transferred  to  the  weighted  scores. 


The  results  of  the  trade  study  presented 
in  Figure  8-9  illustrate  the  effect  of 
overlapping  ranges  on  the  analysis.  The 
preferred  alternative  cannot  be  determined 
in  this  example  since  no  single  alternative 
scores  higher  than  all  others  for  every  value 
within  its  range.  Simply  stated,  although  Alt 
3  has  the  highest  absolute  numerical  score 
and  the  highest  average  score,  there  are 
values  of  Alts  4  and  5  that  are  potentially 
higher  than  given  values  of  Alt  3. 

In  the  example  in  Figure  8-9,  trade  study- 
results  allow  only  the  elimination  of  AJts  1 
and  2  from  further  consideration.  Additional 
information  will  be  needed  to  differentiate 
meaningfully  between  Alts  3,  4,  and  5. 
Further  analysis  may  later  show  Alt  3  in  the 
range  of  736  to  740,  Alt  4  in  the  range  of 
742  to  746,  and  Alt  5  in  the  range  of  752  to 
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755,  which  would  indicate  Alt  5  as  the  clearly 
preferred  alternative. 

Where  the  accuracy  limits  of  the 
performance  evaluation  affect  the  decision, 
several  options  are  available  to  the  decision 
maker: 

a.  Delay  the  decision  until  additional 
information  is  available. 

b.  Acquire  additional  dat  i  or  refine  analysis 
to  reduce  uncertainty. 

c.  Review  criteria  and  weights  for 
modification. 

d.  Acquire  insurance/back-up  capability 
(elect  parallel  development  plans). 

83  TRADE-OFF  ANALYSIS 
APPLICATIONS 

In  applying  trade-off  analysis  methods  to 
actual  program  decisions,  several  limitations 
of  the  analysis  methodology  are  encountered. 

As  discussed  in  Section  8.2,  there  is  a 
reluctance  to  use  trade-off  analyses  when 
criteria  are  not  conducive  to  objective 
evaluations.  Studies  including,  for  example, 
parameters  such  as  credibility,  national 
defense,  and  political  saleability  of  various 
alternatives,  are  not  always  seen  as  amenable 
to  a  numerical  performance  measurement 
scheme.  Although  the  use  of  the  trade-off 
analysis  in  these  situations  is  not  as 
straightforward,  the  trade-off  analysis  can 
replace  the  decision  maker’s  intuitive 
balancing  of  these  factors  with  an  explicit 
methodology. 


Trade-off  analysis  is  used  in  source 
selection.  As  part  of  the  source  selection 
plan  included  in  the  solicitation,  evaluation 
criteria  and  the  relative  importance  of  these 
criteria  are  explicitly  stated.  Trade-off 
analyses  used  in  source  selection  must  comply 
with  Federal  Acquisition  Regulation 
requirements,  Program  Manager’s  decisions, 
and  service  administrative  procedures.  The 
program  office  must  document  the  criteria 
and  weighting  information  used,  as  well  as 
evaluation  methods  and  confidence  levels,  as 
part  of  the  source  selection  approval 
documentation.  Further  guidance  on  the 
source  selection  plan  is  given  in  DoDD 
4105.62  [4]. 

The  limitations  of  the  trade-off  analysis 
process  arise  from  three  sources:  1)  the 
validity  of  the  results  is  limited  by  the  quality 
of  data  and  evaluators,  2)  the  selection  of 
criteria,  weighting  schemes,  and  utility  curves 
can  bias  the  analysis,  and  3)  the  number  of 
alternatives  that  can  be  evaluated  is  limited 
by  the  analysis  burden. 

Handling  large  trade-off  analyses  with 
many  alternatives  requires  use  of  a  "trade 
tree."  The  trade  tree  uncouples  an  oversized 
trade-off  analysis  into  several  smaller  studies. 
For  example,  consider  the  choice  of  lifeboats 
for  carrying  200  personnel  (see  Figure  8-11). 
If  each  of  the  4,000  alternatives  needed  to  be 
evaluated  against  each  selection  criterion,  an 
enormous  evaluation  effort  would  be 
required.  Instead,  the  analysis  can  be  broken 
into  four  smaller  studies:  a  capacity  study; 
a  boat  type  study;  an  attachment  mechanism 
study;  and  a  mounting  location  study.  Each 
smaller  study  would  then  involve  evaluating 
its  respective  alternatives  against  all 
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Figure  8-11 

Use  of  Trade  Tree  to  Uncouple  Trade-Off  Analyses 


LIFEBOAT_STUDY 

5  CAPACITIES  (20,  40,  100.  .  .  PERSONNEL) 

20  TYPES  (ALUMINUM,  FIBERGLASS.  .  .) 

1  0  ATTACHMENT  MECHANISMS 
4  LOCATIONS 

4,000  CONFIGURATIONS  TO  BE  COMPARED  WITH  EACH  CRITERION 


LIFEBOAT  STUDY  TRAPE.TREE 


CAPACITY  STUDY  (5  COMPARISONS  WITH  EACH  CRITERION) 
TYPE  STUDY  (20  COMPARISONS  WITH  EACH  CRITERION) 
ATTACHMENT  MECHANISM  STUDY  (10  COMPARISONS  WITH 
EACH  CRITERION) 

LOCATION  STUDY  (4  COMPARISONS  WITH  EACH  CRITERION) 


TOTAL  OF  39  COMPARISONS  WITH  EACH  CRITERION 


applicable  criteria.  As  a  result,  only  39 
comparisons  would  be  needed  for  each 
criterion. 

8.4  TRADE  STUDY  REPORTS 

Trade  Study  Reports  (TSRs)  are  used  by 
all  decision-making  levels  from  the  systems 
engineering  organization  through  the 
government  program  office,  Program 
Manager,  and  headquarters.  TSRs  document 
the  decision  process  and  are  used  to 
correlate  characteristics  of  alternative 
solutions  to  the  requirements  and  constraints 
which  establish  the  selection  criteria  for  a 
specific  trade  study  area.  Each  report 
documents  the  rationale  used  in  the  decision 
process  and  should  present  risk  assessment 
and  risk  avoidance  considerations.  At  a 
minimum,  each  report  should  contain  the 


analysis  result  and  rationale,  a  description  of 
the  alternatives  considered,  the  selection 
criteria  and  weights,  and  the  results  of  the 
sensitivity  analysis.  Report.  The  report 
format  should  be  coordinated  between  the 
program  office  and  the  contractor  by  tailoring 
TSR  requirements  using  the  streamlining 
methodology  addressed  in  Chapter  10  of  this 
guide. 

TSRs  are  important  contributors  to  the 
formal  technical  review  process.  As  with  all 
systems  engineering  documentation,  the  focus 
and  level  of  detail  changes  as  a  program 
moves  through  the  acquisition  life  cycle.  For 
example,  in  conjunction  with  the  System 
Requirements  Review  (SRR),  available  TSRs 
might  cover  system  cost  effectiveness  and  life 
cycle  cost.  During  Preliminary  Design 
Reviews  (PDRs),  trade  studies  might  be 
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equipment  concepts  related  to  specific 
functions  such  as  fail-safe  concepts,  fault 
isolation,  or  target  acquisition.  The  Critical 
Design  Review  (CDR)  may  include  trade 
studies  for  selection  of  manufacturing 
processes.  Trade  studies  can  provide 
valuable  support  to  many  specialty  areas, 
such  as  in  the  evaluation  of  risk  alternatives 
(see  Chapter  15  of  this  guide  for  more 
detailed  discussion  of  risk  analysis  and 
management).  It  is  important  for  all  trade 
studies  on  a  system  to  use  common  criteria 
and  common  relative  weighting  whether  they 
are  for  risk  management,  reliability  and 
maintainability,  integrated  logistic  support,  or 
other  specialty  areas.  This  will  ensure  a 
common  baseline  for  all  decisions. 

8.5  RISK  TEMPLATE:  TRADE  STUDIES 

(DoD  4245.7-M) 

DoD  4245.7-M  [5]  contains  templates 
addressing  sources  of  risk  during  the 
transition  from  development  to  production. 
Each  template  contains  a  description  of  the 
area  of  risk,  an  outline  for  reducing  risk,  and 
a  time  line  that  shows  the  risk  area’s  relation 
to  the  system  acquisition  cycle.  This 
chapter,  as  other  chapters  in  this  guide, 
incorporates  the  template  provided  by  DoD 
4245.7-M  [5].  The  following  is  an  extract 
from  DoD  4245.7-M  [5]  on  the  subject  of 
trade  studies. 

Trade  studies  are  essential  elements  of 
material  acquisition  programs,  not  only  in 
defining  concepts  that  best  meet  mission 
needs,  but  also  in  fine-tuning  selected 
concepts  during  the  design  process.  Concept 
validation  may  not  be  complete  at  the 
beginning  of  Full  Scale  Development; 


however,  there  is  the  expectation  that 
significant  conceptual  problems  can  be 
resolved  during  the  design  process.  In 
addition,  reducing  production  risk  frequently 
is  not  a  trade  study  criterion.  DoD 
4245.7-M  [5]  recommends  the  following 
outline  for  reducing  risk: 

a.  Concepts  representing  new  technology 
untested  in  the  production  environment  are 
validated  fully  before  FSD. 

b.  Trade  studies  durir  g  the  design  process 
are  oriented  towards  reducing  product  risk, 
by  such  means  as  design  simplification, 
design  for  compatibility  with  production 
processes,  design  for  ease  of  both  factory 
testing  and  built-in  test,  and  design  for 
supportability  and  readiness. 

c.  Early  in  the  design  phase,  full 
consideration  is  given  to  standard 
components  that  have  been  developed  and 
can  meet  the  mission  requirements  (such  as 
standard  avionics  or  egress  seats). 

d.  A  quantitative  trade  parameters  list  is 
developed  and  standardized  across  all  design, 
manufacturing,  and  quality  disciplines  as  a 
priority  task  early  in  the  Research, 
Development,  Test  and  Evaluation 
(RDT&E)  program. 

e.  Trade  study  alternatives  are  documented 
and  preserved  formally  in  design  review 
documentation  to  ensure  system  engineering 
traceability  to  design  characteristics 
downstream. 

f.  Production  transition  trade  studies  are 
based  on  design  and  performance  criteria  as 
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weight  factors  for  trade  study  decisions. 

A  bread  spectrum  of  trade  studies  is 
initiated  duiing  the  C/E  phase.  These  trade 
studies  continue  on  into  FSD  as  a  logical 
approach  to  selecting  the  best  design  once 
the  mission  profile  and  design  requirements 
have  been  specified.  The  final  selection  and 
fine  tuning  of  the  design  approach  must 
consider  such  factors  as  producibility  and 
operational  suitability  as  well  as 
performance,  cost,  and  schedule. 

8.6  SUMMARY 

Evaluation  and  decision  occur  at 
many  points  in  each  phase  of  the  acquisition 
life  cycle.  Although  this  chapter  has  focused 
on  one  approach  to  the  trade  study  process, 
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in  reality  there  are  many  approaches  and 
techniques.  The  "best"  approach  for  analysis 
of  logistic  alternatives  to  support  an 
Integrated  Logistic  Support  Plan  (ILSP)  in 
the  D/V  phase  may  be  totally  different  than 
the  trade-off  techniques  best  suited  to 
trading  reliability  for  maintainability,  or  the 
techniques  required  to  develop  a  software 
module.  However,  all  trade  studies  seem  to 
share  certain  desirable  characteristics.  These 
characteristics  are  summarized  in  Figure  8-12 
as  a  management  checklist  for  evaluation  of 
trade  study  planning  and/or  reports.  Trade 
studies  take  place  throughout  the  acquisition 
cycle,  but  effective  baselining  inhibits  endless 
costly  cycles  of  revisiting  confirmed  decisions. 
A  more  comprehensive  discussion  of 
baselining  is  contained  in  Chapter  11  of  this 
guide. 
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Figure  8-12 

Program  Manager's  Checklist  for  Review  of 
Trade-Off  Planning  and  Studies 


1.  ARE  ALL  VIABLE  ALTERNATIVES  BEING  EXPLORED? 

IS  EACH  ALTERNATIVE  CLEARLY  DEFINED? 

HAVE  THE  ALTERNATIVES  BEEN  PRESCREENED?  HOW? 

ARE  AFFORDABILITY  LIMITS  ESTABLISHED?  SOURCES? 

CAN  ALL  OF  THE  SCREENED -OUT  ALTERNATIVES  BE  DEFENDED? 

2.  ARE  SELECTION  CRITERIA  IDENTIFIED? 

ARE  ALL  SIGNIFICANT  CRITERIA  IDENTIFIED? 

DO  THE  CRITERIA  DISCRIMINATE  BETWEEN  ALTERNATIVES? 

ARE  THE  CRITERIA  MEASURABLE? 

HAVE  THE  CRITERIA  BEEN  PRE- APPROVED? 

3.  IS  THE  CRITERIA  WEIGHTING  SYSTEM  ACCEPTABLE? 

ARE  RATIONALES  FOR  CRITERIA  WEIGHTS  EXPLAINED? 

ARE  CRITERIA  WEIGHTS  CONSISTENT  WITH  GUIDANCE? 

ARE  CRITERIA  WEIGHTS  CONSISTENTLY  DISTRIBUTED  IN  THE 
TREE? 

4.  ARE  UTILITY  (SCORING)  CRITERIA  DETERMINED? 

IS  A  DEFENSIBLE  RATIONALE  ESTABLISHED  FOR  EACH 
CRITERION? 

ARE  CRITERIA  DEVELOPED  FROM  OPERATIONAL  MEASURES  OF 
EFFECTIVENESS  WHERE  POSSIBLE? 

DO  ALL  PLANS  USE  THE  SAME  NUMERICAL  SCALE? 

IS  THE  LOCATION  OF  THE  "ZERO  POINT"  EXPLAINED? 

5.  ARE  EVALUATION  METHODS  DOCUMENTED? 

ARE  TEST  DATA  RELIABILITY  ESTIMATES  (CONFIDENCE 
LEVELS )  I NCORPORATED  ? 

ARE  MODELS  VALIDATED?  WHEN?  WHO? 

6.  HAS  SENSITIVITY  BEEN  ESTIMATED? 

ARE  ERROR  RANGES  CARRIED  THROUGH  WITH  WORST-ON-WORST 
CASE  ANALYSIS? 

HAVE  THE  EFFECTS  OF  CHANGES  IN  THE  UTILITY  CURVE 
SHAPES  BEEN  EXAMINED? 

HAVE  RATIONALES  FOR  THE  LIMITS  BEEN  DEVELOPED? 
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CHAPTER  9 

WORK  BREAKDOWN  STRUCTURES 


9.1  INTRODUCTION 

A  work  breakdown  structure  (WBS)  is  a 
product  oriented  family  tree,  composed  of 
hardware,  software,  services,  and  data,  which 
completely  defines  a  program.  The  systems 
engineering  process  plays  the  critical  role  in 
identification  of  the  product  elements  of  the 
WBS.  The  WBS  displays  and  defines  the 
product(s)  to  be  developed  and/or  produced 
and  relates  elements  of  woik  to  be 
accomplished  to  the  end  product.  The  WBS 
is  the  foundation  for: 

a.  Program  and  technical  planning 

b.  Cost  estimation  and  budget 
formulation 

c.  Schedule  definition 

d.  Statements  of  work  and  specification 
of  contract  line  items 

e.  Progress  status  reporting  and  problem 
analysis. 

The  WBS  is  essential  in  providing  the 
capability  for  program  management  office 
(PMO)  to  exercise  technical,  schedule,  and 
financial  control  of  the  program.  Related 
performance  measurement  systems  include 
the  cost/schedule  control  system  criteria 
(C/SCSC),  Cost/Schedule  Status  Report 
(CSSR),  Cost  Performance  Report  (CPR), 


milestone  measurement/  cost  correlation 
(MM/CC),  and  technical  performance 
measurement  (TPM). 

A  WBS  displays  and  defines  the 
product(s)  to  be  developed  or  produced  and 
relates  the  elements  to  each  other  and  to 
the  end  product.  A  WBS  element  is  a 
discrete,  identifiable  item  of  hardware, 
software,  data,  or  service.  During  the 
acquisition  process,  both  the  government  and 
contractor  have  opportunities  to  "tailor"  the 
WBS.  This  tailoring  should  have  the  goal  of 
adding  or  deleting  elements  that  will 
enhance  the  effectiveness  cf  the  WBS  to 
satisfy  both  technical  management  and  cost/ 
schedule  management  objectives.  The  WBS 
serves  as  a  framework  for  the  contractor’s 
overall  management  system  [1].  Four  basic 
types  of  WBS  formats  are  identified  in 
MIL-STD -881 A  [2]  and  shown  in  Figure  9- 
1:  1)  Summary  WBS,  2)  Project  Summary 
WBS,  3)  Contract  WBS  (CWBS),  and  4) 
Project  WBS. 

9.1.1  Summary  WBS 

A  summary  WBS  is  a  structure  in  which 
the  upper  three  levels  of  the  WBS  are 
specified  by  MIL-STD-881A  12).  The 
summary  WBS  has  a  uniform  element 
terminology,  definition,  and  placement  in  the 
family  tree  structure.  Appendices  A  through 
G  of  MIL-STD-881A  {2]  gives  a  three  level 
WBS  for  each  of  the  seven  types  of  materiel 
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Figure  9-1 

Relationship  Among  Types  of  '-VBSs 


items  procured  by  the  Department  of 
Defense  (DoD)  These  summaries  are  for 
guidance  only  and  need  to  be  adapted  to 
the  needs  of  each  program.  The  defense 
materiel  items  cited  are:  1)  aircraft  systems, 
2)  electronics  systems,  3)  missile  systems.  4) 
ordnance  systems,  5)  ship  systems,  6)  space 
systems,  and  7)  surface  vehicle  systems.  The 
three  levels  included  in  a  summary  WBS 
are: 

Level  1  -  The  entire  system,  also  known 
as  a  defense  materiel  item;  e.g.,  the 
Minuteman  1CBM  System,  the  LHA  Ship 
System,  or  the  M-109A1  Self-Propelled 
Howitzer  System.  Level  1  is  usually  directly 
identified  in  the  DoD  programming7  budget 
system  either  as  an  integral  program  element 
or  as  a  project  within  an  aggregated 
program  element. 


Level  2  -  Major  elements  of  the  defense 
materiel  item;  for  example,  a  ship,  an  air 
vehicle,  a  tracked  vehicle,  and  activities  such 
as  systems  test  and  evaluation  (T&E),  and 
data. 

Level  3  -  Elements  subordinate  to  levei 
2  major  elements;  e.g.,  an  airframe,  the 
propulsion  unit,  or  item  of  data  or  services 
such  as  development  test  and  evaluation 
(DT&E)  or  technical  publications. 

The  MIL-STD-881A  [2]  summary  WBS 
for  an  aircraft  system  is  shown  in  the  top 
half  of  Figure  9-2. 

9.1.2  Project  Summary  WBS 

A  project  summary  WBS  is  a  summary 
WBS  that  is  derived  from  MIL-STD-881A 
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[2]  but  is  tailored  to  the  specific  program. 
The  project  summary  WBS  is  also  specified 
to  three  levels  of  detail.  A  sample  project 
summary  WBS  derived  for  a  particular 
program  is  shown  in  the  lower  half  of  Figure 
9*2. 

9.0  Contract  WBS 

The  CWBS  is  the  complete  WBS 
applicable  to  a  particular  contract  or 
procurement  action.  It  will  generally  contain 
the  applicable  portion  of  the  project 
summary  WBS  plus  any  additional  levels  of 
detail  necessary  for  planning  and  control. 

The  CWBS  outlines  program  tasks  and 
establishes  their  relation  to  program 
organization,  configuration  items,  and 
objectives.  It  establishes  a  logical  indentured 
framework  for  correlating  perfromance, 
technical  objectives,  schedule,  and  cost,  and 
ensures  that  all  derivative  plans  contribute 
directly  to  program  objectives.  Development 
of  the  CWBS  eliminates  redundancy  in  task 
efforts  and  forms  the  basis  for  applying  cost 
and  schedule  controls  such  as  C/SCSC. 
Systems  engineering  plays  a  key  role  in  the 
extension  of  the  CWBS  hardware  elements. 
The  dependence  of  hardware  work  package 
extension  on  the  functional  analysis, 
synthesis,  and  trade-off  process  provides 
correlation  and  traceability  of  the  CWBS  to 
system  requirements.  As  an  integrated  data 
system,  the  systems  engineering 
documentation  also  provides  a  common 
interface  between  specialty  engineering 
efforts  (e.g.;  TPM,  risk  management,  and 
integrated  logistic  support  (ILS))  and 
program  level  activities  (e.g.;  project 
planning,  cost/  schedule  management,  and 


engineering  management).  It  also  plays  a 
key  role  in  ensuring  the  correlation  and 
traceability  of  WBS  product  elements. 

9.1.4  Project  WBS 

The  project  WBS  is  the  complete  WBS 
for  the  program.  It  contains  all  WBS 
elements  related  to  the  development  and/or 
production  of  a  defense  materiel  item  and  is 
formed  by  combining  all  the  CWBSs  in  a 
program.  The  project  WBS  may  be 
delineated  to  five  or  six  levels  of  detail,  with 
the  contractor  responsible  for  developing  the 
lower  levels.  A  partial  project  WBS,  to  five 
levels  of  detail,  is  shown  in  Figure  9-3. 

Other  specialized  WBSs  are  used  that 
suit  particular  applications  during  design  and 
development.  For  example,  a  product  WBS 
could  be  derived  from  the  CWBS  for  use  in 
TPM  analysis,  to  select  items  for 
performance  monitoring.  It  would  contain 
only  those  WBS  elements  associated  with 
the  physical  system.  WBS  elements  such  as 
data  and  services  would  not  be  included  in 
the  breakdown.  Needs  unique  to  a 
particular  discipline  will  often  require 
development  of  variants  of  the  basic  WBS 
types.  Possible  derivatives  may  be:  1)  T&E 
WBS,  2)  Systems  Engineering  WBS,  3)  TPM 
WBS,  4)  Production  Engineering  and 
Planning  (PEP)  WBS,  and  5)  Industrial 
Modernization  Incentives  Program  WBS. 
Each  WBS  would  be  formed  by  extracting 
particular  types  of  tasks  or  products  from  an 
existing  project  WBS. 

9.2  WBS  PREPARATION 

The  PMO  builds  a  project  summary’ 
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Figure  9-3 

Hypothetical  Project  WBS  (Partial) 
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WBS  tailored  to  program  objectives  by 
selecting  applicable  elements  from  the 
summary  WBS  in  MIL-STD-881A  [2].  This 
is  usually  developed  at  the  beginning  of  the 
Concept  Exploration/  Definition  (C/E)  phase 
and  is  included  in  the  C/E  request  for 
proposals  (RFP).  From  the  project 
summary  WBS,  individual  CWBSs  are  then 
developed  by  the  contractor(s)  in  compliance 
with  the  government’s  instructions  contained 
in  the  RFP  (a  preliminary  CWBS  is  normally 
part  of  the  contractor’s  proposal).  The  RFP 
contract  line  items  (CLINs),  configuration 
items  (Cls),  contract  work  statement  tasks, 
and  contract  specifications,  are  elements  of 
the  preliminary  CWBS.  During  C/E  contract 
negotiations,  the  PMO  or  contractors  may 
propose  further  changes  to  the  preliminary 
CWBS  to  enhance  its  effectiveness  in 
satisfying  the  objectives  of  the  particular 


acquisition.  The  final  CWBS  will  be 
incorporated  in  the  C/E  contract.  The 
contractor  then  normally  extends  the  detail 
of  the  CWBS  in  order  to  facilitate  in-house 

planning  and  control  required  in  each 
acquisition  phase.  Systems  engineering  plays 
the  key  role  in  the  expansion  of  the  product 
elements  through  the  functional  analysis, 
design  synthesis,  and  trade-offs  conducted  to 
achieve  the  "best"  break-out  of  product 
elements.  The  CWBS  senes  as  the 
framework  for  the  contractor’s  design  work 
management  system  to  provide  summaries  of 
internal  data  that  are  auditable  and 
traceable. 

The  initial  project  summary  WBS  and 
first  CWBS  are  established  at  the  award  of 
the  first  C/E  contract(s).  When  competition 
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is  maintained  with  competing  alternative 
designs,  there  may  be  more  than  one  project 
summary  WBS,  and  there  will  be  more  than 
one  set  of  contract  WBSs  for  a  program.  As 
the  program  develops  and  additional 
contracts  are  awarded,  the  project  WBS 
extends  the  levels  it  addresses,  but  the  top 
level  structure  remains  unchanged,  barring 
major  changes  in  system  concepts. 

Level  commonality  between  the  project 
summary  WBS  and  the  individual  CWBS 
need  not  be  maintained,  provided  that  the 
approved  project  summary  \\BS  element 
nomenclature  and  definitions  are  not 
violated.  Traceable  summarization  of 
individual  CWBS(s)  into  the  approved 
project  summary  WBS  must,  however,  be 
maintained.  The  PMO  incorporates  into  the 
project  WBS  those  levels  of  the  extended 
CWBS(s)  that  it  considers  necessary  for 
program  management  and  other  related 
requirements.  The  formal  project  WBS  is 
completed  prior  to  the  initiation  of 
production. 

The  systems  engineering  process  ensures 
that  as  the  lower  levels  of  the  product 
elements  of  the  WBS  are  developed,  they 
continue  to  satisfy  the  operational  needs 
specified  in  the  system  specification  (Type  A 
specification).  The  systems  engineering 
process  also  ensures  that  any  changes  to  the 
portions  of  the  WBS  under  contractor 
control  are  conducted  using  trade-off 
processes  and  criteria  that  maintain  system 
integrity.  The  only  differences  between  the 
specification  tree,  which  graphically 
represents  the  family  of  specifications  for  a 
program,  and  the  WBS  should  be  primarily 
in  the  level  of  detail,  not  in  the  correlation 
among  product  elements.  Traceability  should 


be  paramount  throughout  the  WBS 
framework. 

93  WORK  PACKAGING 

In  expanding  the  WBS  to  successively 
lower  levels,  the  requirements  for  day-to-day 
task  management  are  balanced  against  the 
performance  reporting  required  by  program 
management,  corporate  management,  and 
the  government.  For  example,  government 
cost  reporting  generally  requires  very  high 
level  reporting,  while  contractor  program 
management  will  often  require  more  detail, 
or  data  structures  that  reflect  internal 
organizational  structures.  The  "best"  WBS 
for  the  government  is  often  not  viewed  as 
the  "best"  from  the  contractor’s  perspective. 
The  requirements  for  this  balancing  are 
inherent  in  the  approved  capabilities/ 
configuration  of  the  contractor’s  cost/ 
schedule  control  system.  The  lowest  WBS 
elements  coincide  with  the  most  efficient  and 
cost-effective  way  of  controlling  technical 
performance,  schedule,  and  cost  of  the 
program. 

WBS  elements  shomd  be  selected  to 
permit  structuring  budgets,  and 
identification/  tracking  of  costs  to  the  level 
required  for  control.  This  is  accomplished 
by  assigning  job  orders  or  customer  orders 
to  the  cost-account  level  for  in-house  effort, 
and  by  structuring  line  items  or  work 
assignments  on  contracts  in  accordance  with 
the  WBS.  Ordinarily,  a  "cost  account"  will  be 
established  at  the  lowest  level  in  the  CWBS 
at  which  costs  are  recorded  and  can  be 
compared  with  budgeted  costs.  Th>>  cost 
account  (WBS  element)  is  a  natural  control 
point  for  cost/schedule  planning  and  control 
of  a  single  organizational  element. 
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Contractors  maintain  records  to  the  work 
package  level.  The  government  normally 
has  access  to  costs  at  the  cost  account  level. 

At  the  lowest  level,  the  effort  is  broken 
into  discrete  work  packages  associated  with 
both  an  organization  and  a  budgeted 
(cost/schedule)  task  as  illustrated  in  Figure 
9-4.  Criteria  for  establishing  an  effective 
work  package  include  the  following: 

a.  Represent  a  specific,  definable  unit 
of  work 

b.  Define  a  unit  of  work  at  the  level 
where  work  is  performed 

c.  Relate  a  unit  of  work  directly  to  and 
as  an  extension  of  a  specific  element  of 
WBS 


d.  Clearly  distinguish  the  work  from 
that  defined  by  other  work  packages 

e.  Assign  a  unit  of  work  to  a  specific 
single  organizational  element 

f.  Identify  a  specific  start-to-completion 
schedule  representative  of  task 
accomplishment  capability 

g.  Relate  work  package  schedules 
directly  to  and  as  an  extension  of  the 
detailed  program  schedule 

h.  Identify  realistic  budgetary/resource 
requirements 

i.  Limit  each  unit  of  work  to  a  relatively 
short  span  of  time 


Figure  9-4 

WBS/Funct iona 1  Integration 
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j.  Identify  specific  accomplishments 
(outputs)  to  result  from  a  unit  of  work  (e.g.; 
reports,  hardware  deliveries,  and  tests). 

Support  tasks  associated  with  a  particular 
hardware  element,  such  as  qualification  tests, 
acceptance  tests,  and  systems  engineering, 
are  included  as  part  of  the  effort  associated 
with  that  hardware  element.  Support  tasks 
pertaining  to  the  overall  system  effort 
(rather  than  individual  hardware  elements 
comprising  the  prime  mission  equipment) 
are  shown  at  level  2  of  the  WBS.  For 
example,  qualification  testing  for  a  fire 
control  radar,  shown  in  Figure  9-3,  would  be 
included  under  the  WBS  element  for  the 
radar  (1320.02)  at  level  5;  however, 
operational  test  and  evaluation  (OT&E) 
involving  the  entire  air  vehicle  is  included 
under  the  WBS  element  for  system  testing 
(2000)  at  level  2. 

The  contractor  assigns  internal  charge 
numbers  for  each  work  package  identified 
in  the  CWBS,  providing  the  detailed  data 
source  used  for  integrated  cost,  schedule, 
and  performance  reporting.  Cost  accounts 
are  correlated  with  demonstrable 
performance  objectives.  At  scheduled 
completion,  task  performance  is  compared 
with  initial  task  objectives.  If  technical 
requirements  are  satisfied,  the  milestone 
completion  is  approved  and  the  budget  value 
is  credited  to  the  cost  account  as  earned 
value.  WBS  elements  not  achieving  required 
performance  levels  are  identified  through 
various  management  techniques  including 
TPM,  risk  analysis,  and  critical  path  analysis. 
Corrective  actions  are  developed  by 
management  to  bring  projected  deficient 
performance  within  acceptable  levels. 


9.4  DOCUMENTATION 

CWBS  inputs  are  described  in  an 
expanded  narrative  in  the  CWBS  dictionary. 
Each  block  or  element  on  the  CWBS 
diagram  is  identified  in  the  CWBS 
dictionary.  Figure  9-5  is  an  example  of  a 
typical  WBS  dictionary.  Generally,  the 
elements  are  assigned  numbers,  listed 
sequentially  in  the  dictionary  with  necessary 
identification,  definition,  objective  of  the 
element,  synopsis  of  the  effort  required,  and 
the  element’s  relationship  to  other  elements. 
The  WBS  element  "Air  Vehicle" ,  depicted  in 
Figure  9-5,  is  identified  by  WBS  number  at 
WBS  level  2.  In  this  example,  the  WBS 
dictionary  also  includes  a  description  defining 
1)  what  constitutes  the  Air  Vehicle  (  e.g.; 
structural  airframe,  installed  engines),  2)  the 
objective  (provide  flyaway  FXX),  3)  the 
documents  which  describe  the  required  effort 
(detailed  specifications),  and  4)  the  Air 
Vehicle’s  associated  lower  level  (level  3) 
elements  (  e.g.;  airframe,  propulsion). 

The  dictionary  ensures  that  the  tasks 
estimated  are  those  contained  in  the 
proposed  CWBS.  The  proposed  CWBS  and 
dictionary  should  be  sufficient  for  contractor 
organizations  to  estimate  costs  and  schedules 
associated  with  accomplishing  their  assigned 
program  tasks. 

Initial  development  of  the  CWBS 
dictionary  is  normally  conducted  by  the 
contractor  as  part  of  the  proposal 
development  effort  for  the  first  development 
contract.  Maintenance,  update,  and  resupply 
of  the  CWBS  dictionary  is  first  specified  as 
a  contractor-assigned  task  by  the  government 
in  a  contract  data  requirements  list  (CDRL) 
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Figure  9-5 

WBS  Dictionary  Format 


PROJECT  /  PROGRAM  DATE 

rxx  CONTRACT  WORK  BREAKDOWN  STRUCTURE 

CONTRACT  NO.  DICTIONARY  SHEET  ToF 

1 


WBS  LEVEL 


ELEMENT  TITLE 


ELEMENT  DESCRIPTION 


ASK  VEHICLE 

The  complete  flyaway  r.XX  for  delivery  to  the  (J.S.  Government.  The  flyaway  fXX  constitutes  the 
structural  airframe,  installed  engines  and  subsystems,  including  mission  peculiar  equipments,  as 
defined  by  the  Detail  Specification  for  Rode]  t-XX  Aircraft  Weapon  System  including  all  attendant 
addendums  and  the  Avionic  Specification. 


WBS  LEVEL 


ASSOCIATED  LOWER  LEVEL  ELEMENTS 


TITLE 


AIRfRAME 
PROPULSION 
AVIONICS  SUBSYSTEM 
FLIGHT  CONTROL  SYSTEM 
INTEGRATION  AND  ASSEMBLY 
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accompanying  that  request  for  proposals. 
Maintenance  and  update  requirements  are 


also  included  on  all  subsequent  requests  for 
proposals. 


9.5  REFERENCES 

1.  DoDD  5010.20,  "Work 
Structures  for  Defense 
Materiel  Items". 

2.  MIL-STD-881A,  "vVork 
Structures  for  Defense 
Materiel  Items". 


Breakdown 
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CHAPTER  10 


STANDARDS/  SPECIFICATIONS 


10.1  INTRODUCTION 

The  basic  critical  output  of  the  systems 
engineering  process  is  a  complete  set  of 
system  technical  requirements,  documented 
broadly  in  a  system  functional  specification 
(Type  A),  This  document  and  expanded 
lower-level  documents  promote  ease  of 
understanding  of  the  process  of  design,  test, 
production,  and  support  of  a  proposed 
technical  solution  to  an  approved  operational 
requirement.  Every  acquisition  program  has 
a  set  of  unique  specifications  that  define  its 
specific  technical  requirements.  These 
documents  incorporate  or  refer  to  many 
government  standards  to  define  items, 
approaches,  or  procedures  which  may  be 
used  in  the  development  and  production 
process.  These  government  standards  are 
employed  to  give  new  programs  the  benefit 
of  previous  technical  experience,  to  promote 
interchangeability  and  commonality,  and  to 
minimize  costs  of  ownership.  Implementation 
must  be  carefully  considered  to  ensure  that 
general  standards/  specifications  represent 
current  technology,  yet  do  not  create 
unnecessary  costs  for  the  program. 

10 2.  SPECIFICATION  S 

Specifications  are  documents  prepared  to 
support  acquisitions  and  to  describe  items 
which  vary  greatly  in  complexity. 
Specifications  form  the  skeleton  around 
which  the  defense  acquisition  process  is  built, 


and  are  necessary  to  satisfy  the  primary 
objective  of  any  procurement  action,  which 
is  to  obtain  required  products  in  the  proper 
quantity,  of  suitable  quality,  in  the  time 
needed,  and  at  the  lowest  possible  price.  [1] 
They  will  establish  requirements  in  terms  of 
both  design  details  and  performance.  There 
are  two  basic  categories  of  specifications: 
general  specifications  and  program  peculiar 
specifications. 

10.2.1  General  Specifications 

General  specifications,  referred  to  as 
military  specifications,  are  controlled  by  the 
Defense  Standardization  and  Specification 
Program  (DSSP)  and  apply  to  all  acquisition 
programs.  They  are  written  to  cover  systems, 
subsystems,  components,  items,  materials, 
products,  or  processes  that  are  intrinsically 
military  in  character.  General  specifications 
represent  a  particular  requirement  at  a 
particular  time  which  can  be  used  over  and 
over  again  on  many  different  acquisition 
programs.  These  specifications  include  1) 
specifications  for  materials,  parts,  and 
processes,  2)  test  criteria  documentation,  and 
3)  management  specifications. 

10.2.2  Program  Peculiar  Specifications 

Program  peculiar  specifications  apply  only 
to  those  products  developed  to  meet  specific 
operational  requirements.  The  basic  forms 
and  types  of  program  peculiar  specifications 
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are  defined  in  MIL-STD-490A  [2]  as 
illustrated  in  Figure  10-1.  There  are  five 
types  of  program  peculiar  specifications:  1) 
system/ segment  specification,  2)  development 
specification,  3)  product  specification,  4) 
process  specification,  and  5)  material 
specification. 

10.2.2.1  System/Segment  Specification 

A  system/  segment  specification  states  the 
technical  and  mission  performance 
requirements  for  a  system  as  an  entity, 
allocates  requirements  to  functional  areas, 
documents  design  constraints,  and  defines 
the  interfaces  between  or  among  the 
functional  areas.  Normally,  the  initial  version 
of  a  system/  segment  specification  is  based  on 
parameters  developed  during  the  Concept 
Exploration/  Definition  (C/E)  phase.  System 
specifications  are  also  referred  to  as  'Type 
A"  specifications. 

10.2.2.2  Development  Specification 

Development  specifications  state  the 
requirements  for  the  design  or  engineering 
development  of  a  product.  The  development 
specification,  as  defined  in  MIL-STD-490A 
[2J,  is  a  document  applicable  to  an  item 
below  the  system  level  which  states 
performance,  interface,  and  other  technical 
requirements  in  sufficient  detail  to  permit  its 
design,  engineering  for  service  use,  and 
evaluation.  Each  development  specification 
shall  be  in  sufficient  detail  to  describe 
effectively  the  performance  characteristics 
that  each  configuration  item  is  to  achieve. 
Development  specifications  are  also  known 
as  'Type  B,”  "Part  I"  of  two  part 
specifications,  or  "design-to"  specifications. 


10.2.23  Product  Specification 

Product  specifications  are  applicable  to 
any  item  below  the  system  level,  and  may  be 
oriented  toward  procurement  of  a  product 
through  specification  of  primarily  functional 
(performance)  requirements  or  primarily 
production  (detailed  design)  requirements.  A 
product  specification  contains:  1)  the 

complete  performance  requirements  of  the 
product  for  its  intended  use,  2)  necessary 
interface  and  interchangeability  characteristics 
(form,  fit,  and  function),  3)  a  detailed 
description  of  the  parts  and  assemblies  of  the 
product,  and  4)  those  performance 
requirements  and  corresponding  tests  and 
inspections  necessaiy  to  ensure  proper 
fabrication,  adjustment,  and  assembly. 
Product  specifications  are  also  referred  to  as 
'Type  C,"  "Part  II"  of  two  part  specifications, 
or  "build-to"  specifications. 

1033.4  Process  Specification 

Process  specifications  are  applicable  to  a 
service  which  is  performed  on  a  product  or 
material.  Examples  of  processes  are  heat 
treatment,  welding,  plating,  packing, 
microfilming,  and  marking.  Process 
specifications  cover  manufacturing  techniques 
which  require  a  specific  or  unique  procedure 
to  achieve  a  satisfactory  result.  Where 
specific  or  unique  processes  are  essential  to 
fabrication  or  procurement  of  a  product  or 
material,  a  process  specification  is  the  means 
of  defining  such  specific  processes.  Normally, 
a  process  specification  applies  to  production, 
but  may  be  prepared  to  control  the 
development  of  a  process.  Process 
specifications  are  also  known  as  'Type  D" 
specifications. 
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Figure  10-1 

Basic  Forma  and  Typas  of 
Program  Paouliar  specifications 


Specification  Forms 

Form  1: 

Prepared  According  to  MIL-STD-490A  or  its 
Equivalent 

Form  la:  Format  exactly  as  specified  in 

the  appropriate  appendix  to 

MIL- STD-4 90 A 

Form  lb:  Limited  Format  Revisions 

Form  2: 

Commercial  Practice  Specification  With 
Supplementary  Military  Requirements 

Form  3 : 

Commercial  Specification  not  Intended  for 
Competitive  Procurement 

Specification  Types 

Type  A: 

System/Segment  Specification 

Type  B: 

Development  Specification 

Type  Bl:  Prime  Item 

Type  B2:  Critical  Item 

Type  B3:  Non-Complex  Item 

Type  B4:  Facility  or  Ship 

Type  B5:  Software 

Type  C: 

Product  Specification 

Type  Clb:  Prime  Item  Function 

Type  Clb:  Prime  Item  Fabrication 

Type  C2a:  Critical  Item  Function 

Type  C2b:  Critical  Item  Fabrication 

Type  C3:  Non-Complex  Item  Fabrication 

Type  C4:  Inventory  Item 

Type  CS:  Software 

Type  D: 

Process  Specification 

Type  E: 

Material  Specification 
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10.2.2.5  Material  Specification 

Material  specifications  are  applicable  to 
raw  materials  (chemical  compound),  mixtures 
(cleaning  agents,  paints),  or  semi-fabricated 
materials  (electrical  cable,  copper  tubing) 
used  in  the  fabrication  of  a  product. 
Normally,  a  material  specification  applies  to 
production  but  may  be  prepared  to  control 
the  development  of  a  material.  Material 
specifications  are  also  known  as  'Type  E" 
specifications. 

10 3  MILITARY  AND  DOD  STANDARDS 

Military  and  Depattment  of  Defense 
(DoD)  standards  are  documents  that 
establish  engineering  and  technical 
requirements  for  processes,  procedures, 
practices,  and  methods  that  have  been 
adopted  as  standard.  They  are  created 
primarily  to  serve  the  needs  of  designers. 
Their  purpose  is  to  control  variability  of 
products  and  processes..  They  include 
materials,  items,  engineering  practices, 
processes,  codes,  symbols,  type  designations, 
definitions,  nomenclature,  test,  inspection, 
packaging  and  preservation  methods  and 
materials,  and  other  standardization  topics. 
The  distinction  between  military  and  DoD 
standards  is  that  DoD  standards  are 
approved  for  use  with  the  metric  system  of 
measurement. 

10.4  HANDBOOKS 

A  handbook  is  a  reference  document 
which  brings  together  procedural  and 
technical  or  design  information  related  to 
commodities,  processes,  practices,  and 
services.  A  handbook  may  serve  as  a 
supplement  to  specifications  or  standards  to 


provide  general  design  and  engineering  data. 
10.5  DRAWINGS 

Drawings  are  referenced  in  many 
standardization  documents  and  supply 
management  records.  Conversely, 
specifications  and  standards  are  often 
referenced  in  drawings  to  identify  the 
materials,  processes,  and  standard  items 
incorporated  in  assemblies  and  equipment. 
The  basic  standardization  documents  for 
drawings  are  DOD-D-IOOOB  [3]  which 
prescibes  the  requirements  for  engineering 
drawings  and  lists  acquired  in  support  of 
DoD  material,  and  DOD-STD-IOOC  [4] 
which  provides  1)  drawing  practices  for  the 
preparation  of  engineering  drawings,  2) 
procedures  for  numbering,  coding  and 
identification  of  drawings,  3)  methods  for 
revising  and  recording  revisions  on  drawings, 
and  4)  requirements  for  the  preparation  of 
associated  lists. 

There  are  three  levels  of  engineering 
drawings.  Level  1  drawings  are  used  primarily 
in  the  design  effort  to  ensure  that  the 
proposed  design  meets  the  stated  operational 
requirements.  These  drawings  are  used  to 
reduce  technological  uncertainties  by 
confirming  that  the  chosen  technology  is 
feasible  and  suitable  for  analytical  evaluation. 
Level  1  drawings  verify  the  preliminary 
design  and  serve  as  the  basis  for  a  specific 
design  approach.  Level  2  drawings  serve  as 
the  basis  for  the  final  design  approach.  These 
drawings  ensure  that  the  operational 
requirements  are  met  with  respect  to 
performance  and  standard  parts.  They  can  be 
used  for  limited  production  of  items  which 
are  suitable  for  field  test,  deployment  and 
logistic  support.  Level  3  drawings  are  similar 
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to  level  2  drawings,  but  are  more  detailed  in 
certain  areas  to  allow  for  a  competitive 
reprocurement  from  another  manufacturer. 
They  provide  the  engineering  data  in  support 
of  quantity  production. 

10.6  PROGRAM  SPECIFICATION 

DEVELOPMENT  PROCESS 

Program  peculiar  specifications,  as 
products  of  the  systems  engineering  process, 
play  an  integral  role  in  the  product 
development  process.  This  relationship  is 
shown  in  Figure  10-2, 

Specifications  established  during  the 
system  acquisition  process  differ  for  each 
type  of  activity.  They  should  state  only  the 
actual  minimum  needs  of  the  government, 
and  should  describe  supplies  and  services  in 
a  manner  that  encourages  competition  among 
qualified  suppliers.  They  should  also  avoid 
restrictive  requirements  that  might  inhibit 
submittal  of  acceptable  alternative  proposals. 

During  the  C/E  phase,  a  draft  system 
specification  (Type  A)  is  prepared  to 
establish  the  system  functional  baseline 
defining  mission  and  technical  requirements. 
The  draft  system  specification  for  the  initial 
solicitation  of  system  concepts  may  be  little 
more  than  a  copy  of  the  operational 
requirement;  e.g.,  Mission  Need  Statement 
(MNS).  Additional  material  provided  in  the 
solicitation  for  system  design  concepts  should 
avoid  specifications  stated  in  terms  of 
equipment;  rather,  it  should  explain  the  need 
in  mission  or  capability  terms,  schedule 
objectives  and  constraints,  project  cost 
objectives,  and  operational  constraints.  By 
the  end  of  the  C/E  phase,  each  contractor 
should  have  prepared  and  submitted  a 


refined  system  specification  stating  the 
technical  and  mission  requirements  of  the 
system  as  an  entity  and  allocating 
requirements  to  functional  areas.  The  system 
specification  should  be  devoid  of  all  details 
that  could  later  inhibit  the  construction  of 
critical  subsystems,  equipment,  and 
components,  or  the  demonstration  of  the 
concept’s  technological  feasibility.  The 
completed  Type  A  specification,  shown  in 
Figure  10-3,  will  be  approved  by  the  Program 
Manager  (PM)  in  conjunction  with  the 
System  Design  Review  (SDR).  Once 
approved,  the  system  specification  establishes 
the  functional  baseline  configuration  for  the 
proposed  system. 

During  the  Concept  Demonstration/ 
Validation  (D/V)  phase,  draft  development 
specifications  (Type  B)  are  developed  to 
establish  an  allocated  baseline  for  each 
designated  configuration  item  (Cl).  These 
specifications  contain  detailed  design 
requirements  and  performance  characteristics 
of  each  hardware  and  computer  software 
configuration  item  (HWCI  and  CSCI 
respectively).  The  development  specifications 
should  reflect  traceability  of  requirements  to 
the  system  specification.  Once  the  system 
specification  is  approved,  the  development 
specifications  can  be  updated  accordingly  to 
reflect  the  current  definition  of  the  system 
and  the  allocation  of  system  requirements  to 
their  specific  functional  areas.  The  draft 
development  specifications  are  updated  and 
authenticated  early  in  the  Full  Scale 
Development  (FSD)  phase,  usually  not  later 
than  the  Critical  Design  Review  (CDR).  For 
hardware,  this  normally  occurs  in  conjunction 
with  the  HWCI  Preliminary  Design  Review 
(PDR);  for  software,  this  normally  occurs  in 
conjunction  with  the  Software  Specification 
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Figure  10-3 

System/  Segment  Specification  Outline 
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Review  (SSR).  The  updated  system 
specification  and  the  series  of  development 
specifications  constitute  the  system  allocated 
baseline  configuration  which  will  constrain 
contractor  efforts  during  FSD.  The 
specifications  should  not  contain  details  that 
would  inhibit  the  important  trade-off  studies 
and  design  evolution  process  vital  to  this 
phase. 

By  the  time  of  CDR,  the  contractor(s) 
should  have  provided  a  final  update  of  the 
development  specifications  and  a  series  of 
draft  product  specifications  (Type  C).  The 
product  specifications  should  provide  the 
detail  necessary  to  permit  economical 
procurement  of  functional  elements  that, 
when  assembled  into  a  system,  will  perform 
as  a  system  in  accordance  with  the  current 
system  specification.  Product  specifications 
are  normally  approved  in  conjunction  with 
the  Physical  Configuration  Audit  (PCA) 
conducted  on  each  Cl;  thereby  establishing 
the  product  baseline  for  each  Cl,  The 
technical  data  package  (TDP)  represents  the 
formally  accepted  drawings  and  specifications 
required  to  produce,  test,  and  accept  the 
various  configuration  items.  This  package 
constitutes  the  product  baseline  configuration 
of  the  system  and  will  be  contractually 
invoked  for  any  procurement/  reprocurement 
of  the  system. 

10.7  SPECIFICATION  TERMINOLOGY 

A  specification  is  a  series  of  requirement 
statements  written  in  clear,  simple  language. 
It  should  not  contain  descriptive  matter  or 
vague,  redundant,  or  ambiguous  statements. 
Nebulous  adjectives  or  modifiers  such  as 
"adequate,"  "excessive,"  or  "moisture 
resistant,"  should  be  avoided.  The  strictest 


requirement  statements  employ  the  word 
"shall,"  indicating  the  need  for  absolute 
compliance  with  the  requirement.  The 
following  phrases  may  be  employed 
alternately  (in  descending  order  of 
compliance): 

a.  "Shall,  where  practical"  (trade-off 
analysis  required  if  alternate  used) 

b.  "Preferred"  or  "should"  (use  of 
alternate  must  be  justified) 

c.  "May"  (contractor’s  selection 
acceptable). 

Use  of  these  terms  allows  the  contractor 
additional  latitude  to  propose  a  cost-effective 
solution.  The  use  of  "will"  is  limited  to  cases 
of  simple  futurity,  such  as  a  declaration  by 
the  government  that  some  item  or  service 
will  be  available  for  use  when  needed.  The 
term  "and/or"  shall  not  be  used  in 
specifications.  Specifications  shall  not  include 
management,  procedural,  or  statement  of 
work  (SOW)  items  such  as  quantities, 
schedules,  costs,  warranty  provisions,  or 
disposal  instructions.  Requirements  should 
be  quantified  and  verifiable.  The  need  to 
use  judgment  in  verification,  such  as  "best 
design  practice,"  "smooth,"  "good 
workmanship,"  "clean,"  or  "minimum  bend 
radius,"  shall  be  avoided  as  much  as  possible. 

10.8  ACQUISITION  STREAMLINING/ 
TAILORING 

Acquisition  streamlining  is  the  reduction 
of  acquisition  time  and  cost  by  eliminating 
unproductive  government  requirements  from 
solicitations  and  contracts.  Streamlining  calls 
for  using  the  creativity  of  those  closest  to  the 
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detailed  design  and  manufacturing  process 
in  defining  the  most  cost-effective  contract 
requirements  at  the  most  opportune  time. 
The  DoD  acquisition  streamlining  initiative 
also  calls  for  practical  measures  to  preclude 
untimely,  untailorcd,  and  accidentally 
referenced  application  of  military 
specifications  and  standards;  that  is,  to 
specify  required  results  rather  than  detailed 
"how  to"  procedures  in  solicitations  and 
contracts.  The  objective  of  streamlining  is  to 
clearly  communicate  what  is  required  in 
functional  performance  oriented  terms  at  the 
beginning  of  development  and  to  allow 
flexibility  for  the  application  of  the 
contractor’s  experience  and  judgment.  The 
contractor  is  required  by  contract  to 
recommend  detailed  specifications,  standards, 
and  requirements  which  should  be  applied  as 
the  weapon  system  evolves  toward  FSD  and 
eventually  production.  In  this  way,  the 
application  and  tailoring  of  specifications  and 
standards  become  an  integral  part  of  the 
design  process. 

There  are  more  than  40,000  military 
specifications  and  standards  in  the  DSSP. 
The  average  age  of  these  military 
specifications  is  more  than  11  years. 
Cost-effective  application  of  the  pertinent 
portions  of  these  documents  to  a  system 
acquisition  program  should  be  an  integral 
part  of  the  design  and  development  process. 
Too  frequently,  however,  these  documents 
are  invoked  in  boiler  plate,  blanket  fashion 
early  in  the:  program,  so  that  they 
prematurely  constrain  contractor  systems 
engineering  activities.  Documents  are  also 
invoked  implicitly  by  reference  in  the  primary 
document  (automatic  tiering).  For  example, 
Document  A  specified  by  contract  may 
invoke  Documents  B,  C,  and  D  by  reference, 


and  they  in  turn  will  invoke  Documents  E,  F, 
G,  and  H.  Premature  constraint  of 
contractor  activities  inhibits  trade-offs  that 
may  be  necessary  to  achieve  overall  systems 
engineering  objectives  such  as  affordability, 
producibility,  reliability,  and  supportability. 
Both  over-use  of  military  specifications  and 
automatic  tiering  may  contribute  to 
suboptimum  designs  and  unnecessary 
escalation  of  acquisition  costs.  A  number  of 
factors  have  contributed  to  this  problem  on 
the  parts  of  both  the  government  and  the 
contractor: 

a.  Over-conservative  approach  to  design 
engineering 

b.  Failure  to  challenge  requirements  during 
design  and  development 

c.  Follow-on  production  using  the  original 
system  TDP  which  was  adequate  at  the  time, 
but  may  no  longer  be  cost-effective  due  to 
new  technology  and  revised  operational 
requirements 

d.  Lack  of  adequate  technical  data  to  assess 
over-specification. 

The  traditional  approach  to  tailoring 
specifications  and  standards  docs  not  take 
into  consideration  the  timing  of  the  tailoring 
task.  The  application  and  tailoring  of 
military  specifications  and  standards  is 
basically  a  development  and  design  issue. 
Selection  and  tailoring  of  military 
specifications  and  standards  should  be  a 
product  of  the  systems  engineering  process 
rather  than  an  element  of  production 
contract  definition.  During  C/E  and  D/V,  the 
contractor  has  the  systems  engineering 
manpower  base  to  effectively  handle  the 
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labor-intensive  task  of  tailoring. 

The  traditional  problems  which  occur 
when  tailoring  is  done  exclusively  by  the 
government  include: 

a.  Insufficient  time  during  RFP  preparation 

b.  Insufficient  manpower,  skills,  and 
information 

c.  Functional  organization  bias  and 
inclination  to  specify  design  solutions 

d.  Desire  to  minimize  risks  by  maximizing 
requirements 

e.  Lack  of  contractor  incentives  to  assist 
or  question  the  program  office  tailoring 
decisions 

f.  If  the  contractor’s  proposal  includes 
changes  to  the  use  or  content  of  military 
specifications  and  standards,  contractor 
concern  that  it  will  be  perceived  as 
non-responsive,  or  that  it  will  disclose 
information  that  could  compromise  its 
competitive  position. 

Imposing  unnecessary  or  untailored 
requirements  contractually  may  lead  to  poor 
discipline  regarding  compliance  with  contract 
requirements  in  general.  As  a  result, 
compliance  with  truly  necessary  requirements 
may  be  ignored  by  either  the  program  office 
or  the  design  contractor.  Effective 
streamlining,  on  the  other  hand,  leads  to  the 
development  of  fewer  and  more  effectively 
defined  contract  requirements  which  in  turn 
may  foster  disciplined  adherence  to  all 
contract  requirements.  In  addition, 


acquisition  streamlining  seeks  to  cut  down  on 
government  requirements  for  data  (by 
limiting  data  requirements  to  those  actively 
determined  to  be  necessary  for  technical 
management  purposes).  Chapter  14  of  this 
guide  describes  4  he  methodology  for  selecting 
"essential  documentation"  in  lieu  of 
"everything  possible." 

An  acquisition  streamlining  approach  has 
been  outlined  in  DoDD  5000.43  [5],  It  is 
intended  to: 

a.  Utilize  contractor  ingenuity  and 
experience  in  arriving  at  cost-effective 
designs,  while  retaining  government  PM 
decision-making  authority 

b.  Support  the  basic  requirement  to  pursue, 
throughout  the  system  development  process, 
a  design  that  is  economically  producible  as 
well  as  operationally  suitable  and 
field-supportable 

c.  Ensure  development  of  complete  and 
definitive  production  data  and  specifications, 
while  providing  adequate  flexibility  to  the 
contractor  to  optimize  the  systeiu  design. 

DoD  guidance  on  streamlining,  including 
DoDD  5000.43  [5]  and  DoD-HDBK-248  [6], 
has  been  developed  to  facilitate 
implementation.  The  following  are 
procedures  regarding  contractual  referencing 
aspects  of  the  streamlining  initiatives  [7]: 

a.  At  Milestone  0,  specify  system-level 
requirements  in  mission  performance  terms. 
Prior  to  FSD,  military  specifications  and 
standards  shall  be  cited  for  guidance  only. 
These  documents  shall  be  evaluated  and,  if 
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required,  tailored  for  application  to  FSD. 

b.  For  FSD  contracts,  limit  contractual 
applicability  of  specifications,  standards,  and 
related  documents  to  those  cited  in  the 
contract,  and  to  specified  portions  of 
documents  directly  referenced  by  those  cited 
(first-tier  references).  All  other  referenced 
documents  (second-tier  and  below)  shall  be 
for  guidance  only,  unless  specifically  called 
out  in  the  contract. 

c. .  For  production  contracts,  those 
specifications,  standards,  and  referenced 
documents  comprising  the  baseline  for 
production  shall  be  considered  contractual 
requirements  for  procurement  and 
reprocurement  purposes.  Acquisition 
streamlining  should  continue  throughout  the 
production  phase,  with  emphasis  on  ensuring 
that  only  essential  production  and  data 
requirements  are  carried  forward  into 
follow-on  production  contracts. 

d.  When  there  is  a  decision  to  use  items 
already  developed,  such  as  standard  parts 
and  off-the-shelf  items,  all  specifications  and 
standards  that  define  the  product  baseline  for 
the  items  are  contractually  applicable, 
irrespective  of  the  acquisition  phase. 

Previously,  recommendations  for  tailoring 
of  specifications/  contract  requirements  by 
the  contractor  were  optional.  However, 
DoDD  5000.43  [5]  now  stipulates  that 
contractors  be  required,  under  the  terms  of 
the  contract  for  a  particular  phase,  to 
propose  recomendations  for  application  and 
tailoring  of  specifications/  contract 
requirements  in  the  next  phase.  This  activity 
must  become  an  integral  part  of  systems 
engineering  plans.  As  system  design  evolves 


through  D/V,  lower-tier  specifications  and 
standards  will  normally  be  selected  and 
tailored  for  the  next  phase.  For  example, 
identified  requirements  should  be  reviewed 
by  systems  engineering,  tailored  as 
appropriate,  and  identified  as  requirements 
in  the  FSD  proposal.  During  FSD,  a  primary 
task  should  be  to  review  and  scrub  lower-tier 
references  to  ensure  that  manufacturing  and 
process  standards  are  cost-effective. 
Normally  this  process  would  be  completed  by 
CDR.  The  government  PM  will  make  the 
final  determination  as  to  which  contract 
requirements  should  apply  to  the  production 
phase.  The  checklist  shown  in  Figure  10-4 
can  be  used  by  the  PM  in  reviewing  data 
requirements  statements,  specifications,  and 
standards  for  incorporation  in  contracts. 

The  directive  [5]  further  specifies  the  use 
of  the  contractor’s  management  systems, 
internal  procedures,  data  formats,  etc.,  unless 
the  program  office  determines  that  these  do 
not  meet  program  needs.  This  increased 
emphasis  on  contractor  systems,  procedures, 
and  documents  increases  the  contractor’s 
flexibility  in  generating  program 
documentation  in  the  most  efficient  and 
effective  manner.  Use  of  contractor 
management  systems  will  be  effective  only  if 
each  program  office  anticipates  the 
requirement  for  timely,  careful  review  of 
proposed  procedures  and  formats. 

10.8.1  Application  and  Tailoring  of  General 
Procurement  Standards 

Procedures  and  policy  for  the  Defense 
Standardization  and  Specification  Program 
(DSSP)  are  promulgated  by  DoDD  4120.3 
[8].  Specifications,  standards,  handbooks, 
and  other  engineering  documentation 
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prepared  under  the  DSSP  are  intended  to 
state  only  the  actual  needs  of  the  government 
in  a  manner  that  will  encourage  maximum 
competition.  The  objectives  of  the  DSSP 
'defined  by  DoD  4120.3-M  [9])  are  to 
accomplish  the  following: 

a.  Improve  the  operational  readiness  of 
the  military  services 

b.  Conserve  money,  manpower,  time, 
facilities,  and  natural  resources 

c.  Optimize  the  variety  and  minimize 
variability  of  items  (including  subsystems), 
processes,  and  practices  used  in  acquisition 
and  logistic  support 

d.  Enhance  interchangeability,  reliability, 
and  maintainability  of  military  equipment  and 
supplies 

e.  Ensure  that  products  of  requisite  quality 
and  minimum  essential  need  are  specified 
and  obtained 

f.  Ensure  that  specifications  and  standards 
are  written  to  facilitate  tailoring  of  prescribed 
requirements  to  the  particular  need 

g.  Ensure  that  specifications  and  standards 
imposed  in  acquisition  programs  are  tailored 
to  reflect  only  particular  needs  consistent 
with  mission  requirements. 

The  application  of  general  military/  DoD 
specifications  and  standards  to  program 
unique  requirements  is  an  essential  element 
in  transferring  the  accumulated  knowledge  of 
the  military  environment  to  new  programs. 
It  can  greatly  reduce  the  repetition  of 
development  efforts  and  tests  that  are  fully 


provided  in  the  referenced  standards.  Great 
care  must  be  exercised,  however,  to  ensure 
that  the  standards  are  properly  applied  to 
suit  the  stage  of  program  maturity  and  that 
these  requirements  are  adequate  and 
pertinent  to  the  needs  of  the  program. 

In  addition  to  the  specific  program  office 
review  of  standards  as  part  of  the  acquisition 
streamlining  program  described  in  the 
previous  section,  DoD  has  also  established 
procedures  for  continued  review  and  revision 
of  standardization  documents,  including  a 
mandatory  review  of  all  documents  that  have 
not  been  amended,  revised,  or  validated  for 
a  period  of  five  years.  However,  update  may 
involve  intervals  considerably  longer  than  five 
years.  Both  government  and  industry  are 
responsible  for  ensuring  that  each 
procurement  standard  imposed  during 
development  is  both  suitably  tailored  and 
current. 

There  are  a  number  of  appropriate  ways 
to  tailor  procurement  standards.  The 
application  of  a  standard  may  be  limited  to 
specified  components  or  types  of  components 
(for  example,  airborne  electronics)  within  the 
system  by  specifying  the  limits  in  the  body 
of  the  system  specification.  Applicable 
portions  of  a  standard  may  also  be  extracted 
for  incorporation  into  the  text  of  a 
development  specification.  In  either  case,  a 
referenced  standard  may  be  supplemented  by 
descriptive  text  in  the  specification  which 
clarifies  the  intended  requirements  or 
application.  Inapplicable  portions  of  the 
standard  may  be  deleted  by  identifying  them 
in  an  appendix  to  either  specification. 

In  the  application  and  tailoring  of 
standards,  the  order  of  precedence  is  (highest 
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to  lowest): 

a.  Specifications  (federal,  military,  program 
peculiar) 

b.  Standards  (federal,  military,  industry) 

c.  Handbooks  (governmental). 

10.9  RISK  TEMPLATE:  DATA 
REQUIREMENTS 

Procurement  of  excessive  technical  data 
increases  the  risk  of  cost  overruns.  Use  of 
data  procured  by  different  government 
functional  organizations  without  program 
office  coordination  often  results  in  redundant 
requirements  on  the  contractor.  It  is 
estimated  that  direct  costs  for  data  range 
from  6  to  20  percent  of  contract  value,  not 
including  the  overhead  costs  and  the  cost  to 
the  government  to  process,  review,  and 
manage  the  data. 

A  corollary  problem  is  failure  to 
effectively  use  data  for  program  control. 
Control  of  data  requirements,  in  the  past,  has 
been  sporadic.  Even  though  the  problem  of 
poor  data  management  has  been  identified 
in  various  studies  over  the  past  20  years,  it 
receives  little  emphasis  where  there  is  a  lack 
of  top  level  commitment.  DoD  4245.7-M 
[10]  makes  the  following  recommendations 
for  reducing  program  risk: 

a.  All  procurement  data  requirements 
should  be  reviewed  by  an  effective  data 
review  board  before  contract  award  to  ensure 


that  the  data  received  will  satisfy  the 
government’s  needs,  is  in  a  format  suitable 
for  customer  use,  and  is  not  redundant. 

b.  An  integrated  data  management  system 
should  be  established,  both  in  government 
and  industry,  for  each  major  procurement. 
The  objective  of  this  system  is  to  tailor  the 
technical  data  requirements  to  the  needs  of 
each  program. 

c.  Electronic  data  transfer  should  be  used 
to  allow  access  to  pertinent  data  required  by 
the  government.  The  requested  data  can 
then  be  exercised  in  the  government’s  data 
base  to  extract  the  required  information. 

d.  The  data  requirements  for  a  major 
program  should  be  reviewed  at  a  level  high 
enough  to  ensure  that  redundant  data  are 
not  being  requested  by  the  different 
disciplines  within  the  program  office  and  its 
functional  support  organizations. 

e.  Technical  data  libraries  should  be 
established  for  ease  of  data  retrieval,  and  the 
data  should  be  kept  current. 

f.  Data  requirements  should  be  reviewed 
during  each  phase  of  the  program  to  ensure 
that  data  being  procured  meet  the  needs  of 
that  particular  program  phase. 

g.  Data  should  be  procured  using 
well-defined  data  requirements  lists, 
reasonable  cost  estimates,  and  realistic 
schedules. 
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CHAPTER  11 


CONFIGURATION  MANAGEMENT 


11.1  INTRODUCTION 

Configuration  management  (CM)  is  an 
integral  part  of  the  systems  engineering 
management  process  for  system  definition 
and  control.  Its  role  is  1)  to  identify  the 
functional  and  physical  characteristics  of 
selected  system  components,  designated  as 
configuration  items  (CIs),  during  the  system’s 
acquisition  life  cycle;  2)  to  control  changes 
to  those  characteristics;  and  3)  to  record/ 
report  change  processing  and 
implementation  status.  CM  is  thus  the 
means  through  which  the  integrity  and 
continuity  of  the  design,  engineering,  and 
cost  trade-off  decisions  made  between 
technical  performance,  producibility, 
operability,  testability,  and  supportability  are 
recorded,  communicated,  and  controlled  by 
program  and  functional  managers  [1].  One 
of  the  rewards  of  an  effective  CM  process  is 
improved  supportability,  including  updated 
technical  manuals/  documentation,  identified 
spares,  identical/  interchangeable  equipment, 
and  known  configuration. 

At  any  given  time,  CM  can  supply 
current  descriptions  of  developing  hardware 
configuration  items  (HWCI),  computer 
software  configuration  items  (CSCI),  and  the 
system  itself.  CM  provides  traceability  to 
previous  baseline  configurations  of  the 
system  and  for  each  of  the  CIs.  CM  also 
contains  complete  information  on  the 
rationale  for  configuration  changes,  thus 


permitting  analysis  and  correction  of 
deficiencies  when  they  arise.  CM  involves 
four  distinct  functions  :  1)  Configuration 

Identification;  2)  Configuration  Control;  3) 
Configuration  Status  Accounting;  and  4) 
Configuration  Audits  [2]. 

CM  can  be  initiated  by  inputs  from  the 
systems  engineering  process  as  early  as  the 
Concept  Exploration/  Definition  (C/E)  phase 
and  continues  throughout  the  acquisition  life 
cycle  as  the  system  develops  and  is  modified. 
Configuration  changes  occur  throughout  the 
life  of  the  system  as  1)  more  knowledge  of 
the  system  design,  operation,  and 
maintenance  concepts  is  gained;  2)  mission 
requirements  change;  or  3)  non-technical 
factors  such  as  cost  and  schedule  influence 
the  design.  These  changes  must  be 
controlled  to  ensure  first  that  they  are 
cost-effective,  and  second  that  they  are 
properly  documented  so  that  all  users  are 
aware  of  the  current  configuration  status. 

11.2  ESTABLISHING  THE  BASELINE 
CONFIGURATION 

One  of  the  more  important  aspects  of 
CM  is  the  concept  of  baseline  management. 
Baseline  management  is  formally  required  at 
the  beginning  of  an  acquisition  program. 
The  Joint  Services  Regulation  on  CM  [3] 
implements  DoDD  5010.19  [2]  and  defines 
the  baseline  configuration  to  which  CM  is  to 
be  applied. 
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11.2.1  Functional  Baseline 

The  system  functional  baseline  is 
established  with  the  approval  of  the  the 
system  level  specification  (Type  A 
specification)  which  defines  the  technical 
portion  of  the  program  requirements. 
Ideally,  the  system  specification  should  be 
approved  by  the  end  of  C/E,  but  normally  is 
approved  during  the  Concept 
Demonstration/  Validation  (D/V)  phase  for 
major  weapon  system  programs.  An  output 
of  C/E  activity  is  a  draft  system  specification 
which  may  be  approved  by  the  government 
at  that  time  or  used  as  an  initial  system 
specification  in  the  request  for  proposal 
(RFP)  package  for  the  D/V  effort  and 
approved  during  D/V.  The  system 
specification  provides  the  basis  for 
controlling  the  system  design  during  the 
system’s  life  cycle  and  once  approved,  formal 
configuration  control  is  initiated. 

11.2.2  Allocated  Baseline 

An  allocated  baseline  is  established  for 
each  Cl  with  the  approval  of  their  individual 
development  specification  (Type  B 
specification).  These  specifications  define 
the  particular  performance  requirements  for 
each  individual  Cl.  They  are  normally 
developed  during  D/V  and  approved  early  in 
the  Full  Scale  Development  (FSD)  phase. 
The  development  specifications  incorporate 
the  technical  approaches  developed  to  satisfy 
the  objectives  of  the  system  functional 
baseline.  These  objectives  are  translated 
through  the  systems  engineering  process  into 
subsystem  and  Cl  performance  requirements. 
Initial  development  specifications  for  each  Cl 
are  included  in  the  RFP  for  the  FSD  effort 
and  approved  during  FSD.  The  development 


specification  establishes  the  allocated 
baseline  for  a  Cl  and  provides  the  basis  for 
detailed  design  and  development  of  that  Cl 
during  FSD. 

11.23  Product  Baseline 

The  product  baseline  is  established  for 
each  Cl  with  the  approval  of  their  individual 
product  specification  (Type  C  specification), 
including  associated  process  and  material 
specifications  (Types  D  and  E  specifications 
respectively),  engineering  drawings,  and 
detailed  design  documentation.  The  product 
baseline  establishes  the  requirements  for 
HWCI  fabrication  and  CSCI  coding 
respectively.  These  Cl  product  baselines  are 
the  basis  of  the  production  RFP  and 
subsequent  statements  of  work  (SOW). 
The  product  baseline  is  verified  by  successful 
completion  of  the  Functional  Configuration 
Audit  (FCA)  and  Physical  Configuration 
Audit  (PCA). 

113.4  Configuration  Control  Board 

During  development,  the  government’s 
Configuration  Control  Board  (CCB)  is 
responsible  for  reviewing  and  issuing  changes 
to  the  configuration  baseline.  The  CCB 
reviews  all  Class  I  Engineering  Change 
Proposals  (ECPs)  to  determine  if  a  change 
is  needed  and  to  evaluate  the  total  effect  of 
the  change.  The  CCB  typically  consists  of  a 
representative  from  each  of  the  following 
organizations:  Chairman-  Program 

Management  Office  (PMO)  representative, 
User  Command(s),  Engineering,  Training, 
Logistics,  Procurement/  Contracts, 
Configuration  Management  (Secretariat), 
Manufacturing/  Production,  and  others  as 
required. 


11-2 


Within  the  contractor’s  organization,  CM 
functions  are  normally  assigned  to  a 
configuration  manager.  The  contractor  may 
also  have  its  own  internal  CCB  with 
responsibility  for  screening  changes  prior  to 
government  review. 

113  CONFIGURATION  MANAGEMENT 
PRACTICES 

An  excellent  description  of  CM  practices 
can  be  found  in  AFSCP  800-7  [4].  The 
following  paragraphs  are  based  on  that 
document. 

113.1  Configuration  Identification  and 
Status  Accounting 

Configuration  identification  is  the  family 
of  specifications  and  drawings  that  describes 
the  system  or  Cl  during  the  design/ 
development  cycle.  The  identification 
becomes  more  precise  as  the  design 
progresses  toward  production.  This  family  of 
documents  provides  the  basis  for 
development,  testing,  production,  delivery, 
operation  and  support  throughout  the  total 
syste.  I  life  cycle.  CIs  are  identified 
througl  -ocation  of  the  system  specification 
requires  ts  into  lower  tier  requirements 
that  subsequently  become  the  technical 
performance  development  specifications  for 
each  Cl.  Derision  of  a  system  into 
configurate  items  is  a  technical 
management  decision.  In  other  words,  it  is 
an  acknowledgement  that  one  item  should 
be  managed  differently  than  another. 
Selection  of  CIs  is  a  matter  of  judgment. 
Guidelines  for  Cl  selection  are  given  in 
Appendix  XVII  of  MIL-STD-483  [1].  Each 
Cl  should  be  produced  by  a  single  contractor 
and  tested  as  an  entity.  The  program  office 


should  also  limit  the  number  of  CIs  in  order 
to  control  the  management  effort  and  reduce 
cost  to  the  government. 

Once  the  CIs  are  identified,  the 
specifications  defining  them  can  be 
produced.  The  Cl  identification  function 
must  ensure  that: 

a.  All  technical  documentation  describing 
the  functional  and  physical  characteristics  of 
CM  items  is  completely  defined. 

b.  Verified  technical  documents  defining 
the  baseline  are  current,  approved,  and 
available  for  use  when  needed. 

The  Cl  identification  number  provides  a 
permanent  reference  number  for  all  CIs  in 
a  given  type,  model,  or  series.  Part  numbers 
are  usually  needed  down  to  the  throw-away 
components  of  the  lowest  repairable  item. 

Configuration  status  accounting  is  a 
management  information  system  that 
provides  traceability  of  configuration 
identification  and  changes  thereto,  and 
facilitates  the  effective  implementation  of 
changes.  It  consists  of  reports  and  records 
documenting  actions  due  to  changes  that 
affect  the  Cl.  The  basic  documentation 
includes  the  Configuration  Identification 
Index,  describing  the  approved  configuration, 
and  the  Configuration  Status  Accounting 
Report,  describing  the  current  configuration. 
Standardization  of  data  elements  with  regard 
to  format,  frequency,  and  record  keeping,  is 
defined  in  MIL-STD-482  [5]. 

1133  Configuration  control 

Changes  to  CIs  can  only  be  effected  by 
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a  duly  constituted  CCB,  as  described  in 
paragraph  11.2.4.  The  CCB  first  defines  a 
baseline  comprising  the  specifications  which 
govern  development  of  the  Cl  design. 
Proposed  changes  to  this  design  are 
classified  as  either  Class  I  or  Class  II 
changes.  Class  I  changes  affect  form,  fit,  or 
function.  However,  other  factors  such  as 
cost  or  schedule  can  cause  a  Class  I  change. 
A  non-exclusive  list  of  potential  items  is 
provided  in  Figure  11-1  (see  also  Section  4 
of  DoD-STD-480  [6]).  Class  I  changes  must 
be  approved  by  the  Government  prior  to 
being  implemented  by  the  contractor. 


All  other  changes  are  Class  II  changes. 
Examples  of  Class  11  changes  are  editorial 
changes  in  documentation  or  hardware 
changes  (such  as  material  substitution), 
which  do  not  qualify  as  Class  I  changes. 
Government  concurrence  generally  is 
required  in  order  for  the  contractor  to 
implement  Class  II  changes.  Government 
plant  representatives  (CAS,  NAVPRO, 
AFPRO)  usually  accomplish  this  task. 

Changes  are  prioritized  as  "emergency, 
urgent,  or  routine"  as  defined  in 
DoD-STD-480  [6],  according  to  the 


Figure  11-1 

Items  Prompting  a  Class  I  Change 
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criticality  of  the  change.  Emergency  changes 
should  be  processed  within  48  hours,  urgent 
changes  within  30  calendar  days  and  routine 
changes  within  90  calender  days.  Typically 
a  CCB  meets  weekly,  but  meetings  may  be 
convened  on  24-hour  notice  according  to 
need. 

The  configuration  control  process  is 
shown  in  Figure  11-2  and  includes  both 
government  and  contractor  functions.  The 
process  is  initiated  by:  1)  Government 
direction;  2)  Interface  Control  Working 
Group  (ICWG)  activity;  3)  Contractor  design 
or  test  definition  activity;  or  4)  Subcontractor 
action. 

A  change  package  is  first  prepared  that 
includes  the  following: 

a.  Statement  of  the  problem  and 
description  of  proposed  change 

b.  Alternatives  considered. 

c.  Analysis  showing  that  the  change  will 
solve  the  problem. 

d.  Analysis  to  assure  that  the  solution  will 
not  introduce  new  problems. 

e.  Verification  of  interface  compatibility 
including  test,  operations,  safety,  and 
reliability. 

f.  Estimate  of  cost  and  schedule  impact. 

g.  Proposed  specification  or  Interface 
Control  Document  (ICD)  revision. 

h.  Impact  if  not  implemented. 


After  internal  review  by  the  contractor, 
Class  I  changes  are  prepared  in  either  an 
Advance  Change/  Study  Notice  (ACSN) 
(AFSC  Form  223),  a  Preliminary 
Engineering  Change  Proposal  (PECP),  or  an 
Engineering  Change  Request  (ECR). 
Occasionally,  the  change  may  result  in  a 
Request  for  Deviation/  Waiver  (DD  Form 
1694).  Interface  changes  are  defined  on  an 
Interface  Revision  Notice  (IRN).  Following 
approval  of  the  A*^SN  or  ECR  by  the 
government,  a  formal  .iCP  (DD  Form  1692) 
is  developed  containing  Specification  Change 
Notices  (SCNs)  (DE  Form  1696)  and  the 
change  pages,  together  with  supporting  cost 
data.  Upon  ECP  approval  by  the 
government,  the  implementation  plan  is 
issued.  Implementation  status  is  monitored 
by  both  the  government  and  contractor  CM 
organizations. 

Changes  to  released  ICDs  are  processed 
as  defined  for  baseline  changes  with  an 
additional  step.  Each  proposed  ICD  change 
must  be  reviewed  and  approved  by  the 
ICWG  prior  to  approval  of  the  ECP,  which 
allows  incorporation  of  the  change  into  the 
hardware  and  software.  For  that  purpose,  a 
Preliminary  Interface  Revision  Notice 
(PIRN)  approved  by  the  ICWG  will 
accompany  the  ECP.  Approval  of  the  ECP 
approves  the  PIRN. 

113.3  Interface  Management 

Functional  and  performance  interface 
requirements  are  contained  in  the 
appropriate  segment  or  Cl  specifications. 
When  functional  requirements  are  allocated, 
interfaces  may  be  incorporated  in  the  design 
with  full  consideration  for  design  issues. 
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Systems  engineering  and  configuration 
management  personnel  must  coordinate  the 
large  number  of  contractors  and 
organizations  participating  in  the  design 
effort  to  ensure  compatibility  of  all 
interfaces.  For  this  purpose,  interfaces  are 
identified  and  coordinated  by  ICWGs,  which 
are  generally  organized  by  the  prime 
contractor  or  system  integrator,  if  the  latter 
is  a  separate  entity.  A  representative  from 
the  government  PMO  may  chair  the  group, 
although  this  position  is  often  filled  by  the 
prime  contractor,  with  a  program  office 
co-chairman  to  ensure  resolution  of  any 
perceived  contractual  conflicts  that  may  arise 
between  associate  contractors.  The  ICWGs 
may  be  composed  of  several  panels  handling 
specialized  interface  areas  such  as 
electromagnetic  compatibility,  computer 
resources,  and  test  planning.  The  chairman 
organizes  these  groups,  ensuring  that  the 
proper  specialists  are  supported  by 
individuals  with  authority  to  commit  their 
organizations  or  to  obtain  their 
organizations’  approval  for  ICDs  developed 
by  the  ICWG. 

The  nature  of  the  ICD  varies 
considerably,  depending  on  the  interface 
being  documented.  It  can  be  a  physical 
interface,  a  radio  frequency  interface,  or  an 
operational  interface.  Interface  definition 
takes  the  forms  of  drawings,  schematics, 
function  lists,  data  format  diagrams, 
operational  procedures,  equations,  and  other 
data  required  by  the  designers  to  completely 
detail  their  design.  Electrical  circuitry,  for 
example,  is  usually  defined  to  include  the 
first  active  circuit  on  either  side  of  the 
interface.  The  ICD  does  not  duplicate  the 
specification,  rather  it  describes  the  design 
implementation  of  the  requirements  in  the 


specification. 

The  ICD  outline  is  prepared  by  the 
prime  contractor,  and  portions  are  assigned 
to  parties  responsible  for  the  agreement  of 
the  interface  between  system  components, 
including  human  engineering  factors. 
Systems  engineering  must  also  ensure  that 
various  interfaces  are  compatible  or  do  not 
force  unnecessary  costs  on  interfacing 
systems.  Following  completion,  the  ICD  is 
signed  by  all  parties  involved  at  the  interface 
and  is  placed  under  configuration 
management  control.  The  ICD  then  has  the 
same  status  as  a  specification  in  that  it 
represents  the  baseline  configuration,  and 
any  changes  must  be  acted  upon  by  the 
appropriate  CCB.  A  number  of  CCBs  may 
be  involved  in  implementing  an  ICD  change, 
including  CCBs  of  interfacing  contractors  or 
the  government  program  office  that  must 
approve  any  changes  to  the  segment 
interfaces. 

There  is  no  MIL-STD  format  for  the 
development  of  ICDs  similar  to  that  of 
MIL-STD-490  [7]  for  specifications.  A 
sample  ICD  outline  is  provided  in  Chapter 
4  of  this  guide.  Note  that  the  ICD 
addresses  not  only  the  design 
implementations  of  an  interface,  but  also 
covers  items  such  as  exchange  of  models 
(both  physical  and  mathematical),  facilities 
required,  and  availability  of  personnel  for 
special  integrated  operations.  It  also 
identifies  responsibility  for  verification, 
similar  to  specification  practice. 

11 .3.4  Configuration  Audits 

Audits  validate  that  development 
requirements  are  achieved  and  that  product 
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configuration  is  identified  by  comparing  the 
Cl  with  its  technical  documentation.  Two 
kinds  of  audits  are  performed:  1) 

Functional  Configuration  Audit  (FCA)  and 
2)  Physical  Configuration  Audit  (PCA),  as 
described  in  M1L-STD-1521  [8]  and  Chapter 
12  of  this  guide. 

The  FCA  is  a  means  of  validating  that 
development  of  a  Cl  has  been  completed 
satisfactorily  and  that  the  item  functions  as 
required.  It  is  a  prerequisite  to  the  PCA. 
The  FCA  is  normally  performed  during  FSD 
just  prior  to  production. 

The  PCA  is  a  means  of  establishing  the 
product  baseline  as  reflected  in  the  product 
configuration  identification,  and  is  used  for 
the  production  and  acceptance  of  CIs.  The 
PCA  may  be  accomplished  during  FSD; 
however,  it  is  usually  delayed  until  the 
beginning  of  the  Production  phase  so  it  may 
be  accomplished  on  an  early  representative 
production  unit.  A  PCA  is  normally 
required  on  the  first  Cl  to  be  delivered  by  a 
new  contractor  even  though  a  PCA  was 
previously  accomplished  on  a  like-production 
article  delivered  by  a  different  contractor. 

11.4  DOCUMENTATION 

11.4.1  Configuration  Management  Plan 

For  major  systems,  a  Configuration 
Management  Plan  (CMP)  is  usually  required 
to  be  submitted  by  the  contractor  either  with 
the  FSD  phase  proposal  or  early  in  FSD. 
The  plan  content  for  HWCIs  is  outlined  in 
Appendix  I  of  MIL-STD-483  II].  The  plan 
content  for  CSCIs  is  included  in  the 
Software  Development  Plan  (SDP),  the 
Software  CMP,  or  the  System  CMP.  Once 


the  system  is  deployed,  however,  CM 
becomes  the  responsibility  of  the 
government.  The  configuration  of  all  units, 
regardless  of  location,  must  be  known  in 
order  to  ensure  that  changes  and 
modifications  can  be  installed  promptly  and 
properly.  The  government’s  CM?  must 
make  provision  for  data  flow  to  and  from 
deployed  units,  to  ensure  current 
configuration  knowledge.  Appendix  XI  11] 
describes  the  System  Allocation  Document, 
which  provides  the  geographical  location  of 
each  Cl  and  its  installed  equipment. 
Appendix  XV  [1]  defines  the  method  for 
reporting  the  accomplishment  of  changes. 

11.4.2  CCB  Documentation 

CCB  documentation  includes  approved 
Class  l  ECPs  with  change  pages  for  updating 
the  baseline.  In  addition,  the  CCB  will 
distribute  minutes  normally  containing  the 
agenda,  a  list  of  attendees,  a  summary  of 
discussions  on  each  item,  the  disposition  of 
each  agenda  item  with  justification,  change 
package  charts,  and  supporting 
reports/analyses,  as  applicable. 

11.43  Change  Control  Forms 

Change  control  forms  are  the  basic 
means  of  initiating,  evaluating,  approving, 
releasing,  and  implementing  changes.  They 
can  also  be  used  for  reporting  problems, 
requesting  modifications,  and  submitting 
change  proposals.  They  are  a  key  source  of 
information  concerning  the  status  of  changes 
during  change  processing. 

11.43.1  Advance  Change/  Study  Notice 

The  ACSN  (AFSC  Form  223)  may  be 
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used  by  the  contractor  to  provide  the 
government  with  advance  information  of  a 
proposed  change.  It  describes  the  item 
affected,  the  need  for  the  change,  a  brief 
description  of  the  change,  alternatives,  and 
a  rough  cost  estimate.  The  procuring 
activity,  after  reviewing  the  ACSN,  will 
either:  1)  reject  the  change,  2)  request 

additional  information,  3)  direct  an  alternate 
course  of  action,  or  4)  authorize  the 
necessary  engineering  effort  to  prepare  a 
formal  ECP. 

11.43.2  Engineering  Change  Proposal 

The  ECP  is  used  to  propose  Class  I 
changes  to  the  government.  A  preliminary 
ECP  may  be  used  in  place  of  an  ACSN  to 
obtain  government  approval  to  proceed  with 
detailed  engineering  analysis  and  design 
prior  to  incorporation  of  a  change.  The 
ECP  package  contains  a  description  of  the 
change,  justification,  point  of  effectivity, 
effect  on  performance  allocation  and 
interfaces,  impact  on  integrated  logistic 
support  and  operational  effectiveness, 
changes  to  Cl  specifications,  development 
requirements  and  status,  and  results  of 
trade-off  analyses  with  alternative  solutions. 
If  the  program  is  in  production,  additional 
data  on  cost  and  schedule  impact  are 
required.  A  "not-to-exceed"  cost  is 
submitted  with  a  preliminary  ECP.  If 
approved  by  the  government,  a  formal  ECP 
accompanied  by  a  SCN  covering  exact 
changes  to  the  Cl  specification  is  submitted 
to  the  government,  together  with  detailed 
costs  and  schedules.  The  ECP,  DD  Form 
1692,  is  contained  in  DoD-STD-480  [6], 

11.433  Specification  Change  Notice 


The  SCN  is  used  to  propose,  transmit, 
and  record  changes  to  a  specification. 
Initially  it  is  used  to  submit  proposed  Class 
I  specification  change  pages,  accompanied  by 
an  ECP,  for  government  approval.  After  the 
proposed  documentation  change  is  approved, 
the  SCN  is  used  to  transmit  the  change 
pages  to  document  holders.  The  SCN,  DD 
Form  1696,  is  described  in  DoD-STD-480 
[61. 

11.43.4  Request  for  Deviation/  Waiver 

A  Request  for  Deviation/  Waiver  is  used 
to  request  and  document  temporary 
departures  when  permanent  changes  are  not 
acceptable.  The  requests  are  usually 
processed  by  the  CCB.  A  deviation  is 
written  authorization  granted  prior  to 
product  development  to  permit  departure 
from  a  particular  performance  or  design 
requirement  for  a  specified  product  or 
period  of  time.  A  waiver  is  written 
authorization  to  deliver  a  configuration  item 
that  has  been  found,  after  development,  to 
depart  from  specified  requirements,  but  that 
nevertheless  is  considered  suitable  for  use  or 
rework.  The  Request  for  Deviation/Waiver, 
DD  Form  1694,  is  contained  in  DoD- 
STD-480  [6J. 

11.43.5  Interface  Revision  Notice 

The  Interface  Revision  Notice  (IRN)  is 
used  to  propose,  transmit,  and  record 
changes  to  an  ICD.  It  results  primarily  from 
ICWG  meetings,  and  must  first  be 
coordinated  with  and  approved  by  affected 
parties.  The  IRN  is  then  transmitted  to  the 
CCD.  Following  approval  by  the  CCB,  the 
IRN  is  then  used  to  transmit  change  pages. 
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11.5  RISK  TEMPLATE: 

CONFIGURATION  CONTROL 

A  common  source  of  risk  in  the 
transition  from  development  to  production  is 
failure  to  establish  and  maintain  a  strong 
configuration  management  system.  Direct 
application  of  boilerplate  policies  and/  or 
invoking  military  specifications  and  standards 
leads  to  ineffective  control  or  overly  complex 
and  costly  approaches  to  managing 
configuration.  In  a  loosely  implemented  CM 
system,  design  changes  can  occur  without 
proper  maintenance  of  the  configuration 
change  documentation  after  the  baseline  is 
established.  Lack  of  good  CM  systems  leads 
to  many  pitfalls,  including  an  unknown 
design  baseline,  excessive  production  rework, 
poor  spares  effort,  stock  purging  rather  than 
stock  control,  and  inability  to  resolve  field 
problems.  Poor  CM  is  a  leading  cause  of 
increased  program  costs  and  lengthened 
procurement  schedules.  DoD  4245.7-M  [9] 
proposes  the  following  outline  for  reducing 
risk: 

a.  An  effective  configuration  management 
system  should  contain  the  following  features: 

1.  It  should  be  tailored  from  an  effective 
set  of  guidelines  and  standards  to  fit  tne 
nature  of  the  program,  including  hardware 
and  logistics  support  elements. 

2.  Corporate  or  division  policy  should 
recognize  the  importance  of  proper  CM  in 
the  development  of  a  new  program,  and 
emphasize  the  need  to  generate  an  adequate 
plan  for  implementation. 

3.  A  CMP  should  be  streamlined,  yet 
encompass  the  entire  life  cycle  of  the 


program,  recognizing  the  requirements  of 
each  phase  of  the  acquisition  life  cycle  and 
the  complexity  of  the  system  configuration. 

4.  The  CMP  should  establish  the  mode 
of  operation  and  interface  relationships 
among  vendors,  subcontractors,  contractor, 
and  customer. 

b.  The  CM  organization  should  be 
properly  staffed  and  with  individuals  having 
authority  commensurate  with  assigned 
responsibility. 

c.  The  specification  tree,  engineering 
release,  and  drawing  discipline  should  be 
managed  by  documentation  requirements 
established  through  the  CMP. 

d.  The  staff  should  be  trained  in  the 
established  CM  system. 

e.  The  CM  program  should  be  disciplined 
in  order  to  organize  and  implement,  in  a 
systematic  fashion,  the  process  of 
documenting  and  controlling  configuration. 

f.  Status  accounting  systems  should  be 
updated  fr  equently  by  timely  feedback  from 
user  activities. 

g.  Configuration  control  procedures  should 
ensure  the  establishment  and  maintenance  of 
design  integrity. 

h.  Configuration  audits  should  be 
performed  to  establish  the  design  baseline 
and  to  validate  the  drawing  package  before 
production  release. 

i.  Manufacturing  engineering  should 
interface  with  configuration  control  for  work 
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instruction  planning. 

j.  The  transition  from  contractor  to 
government  responsibility  should  be 
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CHAPTER  12 


TECHNICAL  REVIEWS 


12.1  INTRODUCTION 

Technical  reviews  are  an  integral  and 
essential  part  of  the  systems  engineering 
process.  These  reviews  can  range  from  very 
formal  technical  reviews  by  government  and 
contractor  systems  engineers  to  very  informal 
reviews  concerned  with  product  and/or  task 
elements  of  the  work  breakdown  structure 
(WBS)  that  involve  only  a  few  government 
and/or  contractor  personnel.  All  reviews 
share  the  common  objective  of  determining 
the  technical  adequacy  of  the  existing  design 
to  meet  known  technical  requirements.  As 
the  acquisition  program  moves  through  the 
life  cycle,  the  reviews  become  more  detailed 
and  definitive.  Technical  reviews  must  cover 
all  related  engineering  specialty  disciplines. 

Figure  12-1  illustrates  the  relationship  of 
technical  reviews  to  the  system  development 
cycle.  Although  the  systems  engineering 
process  is  iterative  and  cannot  be  precisely 
related  to  acquisition  phasing,  the  scheduling 
of  technical  reviews  depicted  on  Figure  12-1 
indicates  the  logical  time  frame  for  a  ^lven 
system  or  system  segment  by  acquisition 
phase.  The  precise  scheduling  of  all  formal 
technical  reviews  for  each  configuration  item 
(Cl)  or  aggregate  of  CIs  will  normally  be 
included  in  the  System  Engineering 
Management  Plan  (SEMP)  and  in  the 
statement  of  work  (SOW)  of  the  contract. 
The  need  for  formal  reviews  is  controlled  by 


MIL-STD-1521B  [1].  The  requirements  of 
this  standard  should  be  tailored  to  such 
factors  as  program  complexity,  level  of 
inherent  technical  risk,  and  number  of 
participating  contractors.  For  a  non  complex 
system,  some  reviews  may  not  be  required, 
or  if  required  may  be  limited  in  scope.  This 
tailoring  should  result  in  development 
contracts  specifying  application  of  a  tailored 
MIL-STD-1521B  [1]  that  reflects  a  limited 
scope  technical  review  effort. 

The  number  and  depth  of  forma! 
technical  reviews  should  also  be  dependent 
upon  the  Cl  technical  risk  level  (new  design 
versus  commercially  available,  or  the  degree 
of  any  modifications).  For  example,  a  newly 
developed  item  may  require  the  majority  of 
the  review  topics/  items  and  audits  defined 
in  MIL-STD-1521B  [1].  A  commercially 
available  Cl  with  the  appropriate 
documentation  (e.g.,  verified  test  results, 
specifications,  and  drawings)  may  require 
reviews  or  audits  limited  to  its  application  to 
the  program  and  its  interfaces.  In  the  case 
of  modified  designs,  one  must  consider  the 
degree  of  the  modifications  and  effect  on 
technical  risk.  Reviews  and  audits  may  be 
limited  to  the  modifications  and  their 
interfaces. 

The  schedule  for  technical  reviews  and 
audits  is  extremely  important.  If  they  are 
conducted  too  early,  the  item  for  review  will 
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Technical  Reviews  and  the  System  Development  Cycle 

Figure  12-1 


12-2 


not  be  adequately  defined.  Conversely,  if  the 
review  is  too  late,  the  program  commitments 
could  have  been  made  erroneously,  and 
correction  will  be  both  difficult  and  costly.  A 
good  method  for  scheduling  technical  reviews 
is  to  relate  them  to  the  documentation 
requirements.  For  example,  schedule  a 
Preliminary  Design  Review  (PDR)  after  the 
hardware  development  specifications  (Type 
B)  or  software  top  level  design  documents 
and  software  test  plan  are  available. 
Scheduling  of  audits  is  dependent  not  only  on 
documentation  availability,  but  also  on 
hardware/  software  availability  and  the 
completion  of  the  acceptance  qualification 
tests.  The  time  frame  for  reviews  and  audits 
will  vary,  depending  on  the  requirements 
associated  with  a  particular  program.  The 
schedule  for  each  review  or  audit  should  be 
requested  from  the  offeror  as  part  of  the 
proposal,  or  as  part  of  the  SEMP  (which  can 
be  part  of  the  proposal). 

12.2  FORMAL  TECHNICAL  REVIEWS 

Formal  technical  reviews  conducted  in 
accordance  with  MIL-STD-1521B  [1]  include: 

a.  System  Requirements  Review  (SRR) 

The  SRR  is  the  first  major  review.  It  can 
be  either  an  internal  government  or 
government/  contractor  review.  The  review 
will  focus  on  mission  area  analyses  (MAA) 
and  systems  engineering  documentation  ,  in 
particular  the  draft  system  level  specification 
(Type  A). 

b.  System  Design  Review  (SDR) 

This  review  will  normally  serve  as  a  key 
transition  milestone  between  the  Concept 


Demonstration/  Validation  (D/V)  and  Full 
Scale  Development  (FSD)  phases  or  between 
the  system  and  lower  level  development 
activities.  It  will  focus  on  more  detailed 
systems  engineering  documentation,  as  well 
as  the  completed  system  specification.  The 
SDR  provides  a  review  vehicle  for  data  that 
are  used  to  establish  the  system  functional 
baseline. 

c.  Preliminary  Design  Review  (PDR) 

This  review  represents  the  approval  to 
begin  detailed  design.  Its  primary  focus  is  on 
the  adequacy  of  top  level  design 
documentation  for  hardware  and  software 
CIs,  and  the  completeness  of  the 
development  specifications  (Type  B), 
especially  for  hardware  CIs  (HWCIs). 

d.  Software  Specification  Review  (SSR) 

The  SSR  is  conducted  when  the  system 
level  computer  software  requirements  have 
been  defined  and  allocated  to  computer 
software  configuration  items  (CSCIs).  The 
primary  focus  is  on  the  completeness  of  the 
CSCI  development  specifications  (Type  B-5). 

e.  Critical  Design  Review  (CDR) 

The  CDR  is  generally  the  transition  point 
between  detailed  design  and  fabrication  of  a 
Cl  or  aggregate  of  CIs.  Its  primary  focus  is 
on  the  completed  detailed  design 
documentation  and  draft  product 
specifications  (Type  C)  for  HWCIs  and 
CSCIs. 

f.  Test  Readiness  Review  (TRR) 

The  completion  of  TRR  is  the  program 
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WHEN  I  PURPOSE  I  DOCU  MEN  TAT  I  ON/ DATA 


Technical  Review  Summary 
Figure  12-2 
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(Conducted  lncrura«ntu(ly  throuflhout  F80) 


event  coincident  with  the  initiation  of  the 
formal  CSCI  testing.  Its  primary  focus  is  on 
the  adequacy  of  the  software  test  procedures. 

Figure  12-2  identifies  the  principal 
technical  reviews.  More  detail  on  each 
review  is  provided  in  the  following 
paragraphs. 

12.2.1  System  Requirements  Review 

SRRs  are  normally  conducted  during  the 
C/E  or  D/V  phases  after  the  system  level 
functional  analysis  has  been  performed  and 
the  system  level  requirements  have  been 
allocated.  The  purpose  of  the  review  is  to 
ensure  that  system  requirements  have  been 
completely  and  properly  identified  and  that 
there  is  a  mutual  understanding  between  the 
government  and  the  contractor  on  system 
requirements.  Particular  emphasis  is  placed 
on  ensuring  that  adequate  consideration  has 
been  given  to  logistic  support,  software,  test, 
and  production  constraints. 

The  primary  documents  used  in  this 
review  are  products  of  the  functional  analysis, 
synthesis,  and  trade  studies:  functional  flow 
block  diagrams  (FFBDs),  requirements 
allocation  sheets  (RASs),  and  time  line  sheets 
(TLSs),  supplemented  by  MAA,  and  system 
simulations  (e.g.,  relating  to 
survivability/vulnerability  (S/V)  or  reliability/ 
availability/  maintainability  (RAM))  at  the 
most  general  level.  Traceability  of  allocated 
requirements  to  the  capability  of  the  system 
to  meet  the  mission  need  and  program 
objectives  within  planned  resource  constraints 
must  be  demonstrated  by  correlation  of 
technical  and  cost  information  through  the 
WBS.  Specific  items  to  be  reviewed  include: 


a.  MAA 

b.  System  level  FFBDs,  TLSs,  and  RASs 

c.  System  trade  studies  including  system 
effectiveness  and  life  cycle  cost 

d.  Independent  cost  evaluation  (ICE) 

e.  System  interface  studies 

f.  Preliminary  manufacturing  plans 

g.  Manpower  requirements  plans 

h.  Milestone  schedules. 

The  technical  documentation  that  is  the 
subject  of  this  review,  primarily  the  system 
level  specifications  (Type  A),  is  used  to 
establish  the  formal  system  functional 
baseline. 

1222  System  Design  Review 

The  SDR  is  the  final  review  before 
submittal  of  D/V  phase  products.  This  may 
be  the  initial  FSD  phase  review  for  systems 
that  do  not  require  a  formal  D/V  phase  but 
are  sufficiently  complex  to  warrant  formal 
assessment  of  the  allocated  requirements. 

The  focus  of  the  SDR  is  to  evaluate  the 
optimization,  traceability,  correlation, 
completeness,  and  risk  of  the  system  level 
design  to  fulfill  the  system  functional  baseline 
requirements.  The  review  encompasses  the 
total  system  requirements  (e.g.,  operations, 
maintenance,  test,  and  training)  for  hardware, 
software,  facilities,  personnel,  and  preliminary 
logistic  support  considerations.  This  is  a 
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review  of  systems  engineering  activities  that 
yield  products  defining  the  system  level: 

a.  Mission  and  requirements  analysis 

b.  Functional  analysis 

c.  Requirements  allocation 

d.  Manufacturing  methods/  process 
selection 

e.  Program  risk  analysis 

f.  System  effectiveness  analysis 

g.  Logistics  support  analysis  (LSA) 

h.  Trade  studies 

i.  Intra-  and  inter-system  interface  studies 

j.  Integrated  test  planning 

k.  Engineering  specialty  discipline  studies 

l.  Configuration  management  plans. 

A  technical  understanding  is  reached  on 
the  valid:*.  the  de  ;e  of  completeness 
of  the  s>*'~m  level  design,  specification,  and 
acquisition  raid  life  cycle  cost  estimates.  The 
formal  system  functional  baseline  is  normally 
established  at  SDR  once  the  system 
specification  (Type  A)  is  approved. 

12.23  Preliminary  Design  Review 

PDRs  are  conduucd  tor  era.,’:;  i.WCI  and 
CSCI  after  top  level  design  efforts  are 
completed,  but  prior  to  the  start  of  detailed 
design.  They  may  be  held  after  the  approval 


of  the  development  specifications,  or  their 
approval  may  be  accomplished  at  the  PDR, 
which  is  normally  the  case.  Either  way, 
completion  of  the  PDR  represents  approval 
to  begin  detailed  design.  A  design  review  is 
not  considered  formally  complete  until  the 
contracting  agency  provides  formal 
acknowledgement  indicating  approval  or 
contingent  approval  pending  satisfactory 
completion  of  resultant  action  items. 

A  PDR  is  held  for  each  Cl  or  a 
functionally  related  group  of  CIs  prior  to  the 
system  level  review.  Overall  technical  risks 
for  each  item  are  reviewed.  Appropriate 
emphasis  is  given  to  review  of  the 
preliminary  design  of  all  system  elements: 
hardware,  software,  personnel,  facilities,  and 
procedural  data.  Principal  documentation  for 
this  review  is  the  development  specification 
(Type  B)  and  top  level  design  documents. 
Traceability  of  all  specified  technical 
requirements  should  be  demonstrated, 
through  systems  engineering  documentation 
to  the  allocated  baseline.  Typical  items  to  be 
reviewed  include: 

a.  Development  specifications  (Type  B) 

b.  Preliminary  design  synthesis  of 
development  specifications 

c.  Trade  studies  and  design  studies 
supporting  preliminary  design  of  CIs 

d.  Lay-out  drawings  for  CIs 

e.  Engineering  specialty  studies  (e.g., 
RAM,  safety,  integrated  logistic  support 
(ILS),  producibility,  and  human  engineering) 

f.  Interface  requirements  for 


12-6 


developmental  HWCIs 

g.  Mock-ups,  models,  breadboards,  and 
prototype  hardware 

h.  Systems  engineering  documentation 
(e.g.,  FFBD,  RAS,  DS,  and  SBD)  required 
to  support  review  of  the  development 
specifications 

i.  Initial  identification  of  development 
specifications  support  equipment  items  (e.g., 
testing  consoles  and  power  equipment) 

j.  Computer  software  top  level  design 
and  computer  software  test  plan. 

During  the  PDR,  special  attention  is 
directed  toward  interface  documentation,  high 
risk  areas,  long  lead  times,  and  system  level 
trade  studies  that  integrate  preliminary  design 
concepts.  The  government  reviews  the 
progress  of  the  systems  engineering  process 
by  examining  the  description  of  proposed 
system  elements  to  ensure  that  the  system 
design  optimization  trade-offs  fully  integrate 
the  operations,  logistic  support,  test, 
production,  and  deployment  requirements. 
In  instances  where  a  contractor  has  identified 
the  requirement  for  government  furnished 
equipment  (GFE)  in  the  Department  of 
Defense  (DoD)  inventory,  the  government 
will  also  validate  the  availability  of  those 
items. 

12 .2.4  Software  Specification  Review 

An  SSR  for  each  CSCI  will  be  conducted 
after  the  SDR,  but  prior  to  initiation  of  top 
level  design  for  each  individual  CSCI.  These 
reviews  are  keyed  to  the  overall  system 
development  cycle  and  they  must  follow  after 


system  level  hardware/  software  allocation 
decisions  are  made.  An  SSR  may  be 
conducted  prior  to  or  after  a  hardware  PDR 
depending  on  the  relationship  between  a 
particular  CSCI  and  HWCI;  in  all  cases, 
however,  an  SSR  will  be  conducted  before 
the  PDR  for  that  particuler  CSCI. 

The  purpose  of  this  review  is  to  establish 
the  formal  allocated  baseline  for  each  CSCI. 
Emphasis  is  on  demonstrating  the  adequacy 
of  the  software  requirements  specification, 
interface  requirements  specification,  and  the 
operational  concept  document. 

12.2.5  Critical  Design  Review 

A  CDR  is  conducted  for  each  HWCI  and 
CSCI  before  release  of  design  for  fabrication. 
For  CSCIs,  the  CDR  is  conducted  before 
coding  and  informal  software  testing.  For 
large,  complex  systems,  the  CDR  may  be  a 
progressive  or  incremental  review  of  CIs 
culminating  in  a  system  level  CDR  that 
essentially  reviews  the  completeness  of 
preceding  CDRs  and  ensures  adequate 
interfaces. 

During  the  CDR,  the  detailed  design  for 
each  Cl  is  disclosed  in  the  form  of  a  draft 
product  specification  (Type  C)  and  related 
engineering  drawings.  The  approved  detailed 
design  from  this  review  serves  as  a  basis  for 
final  production  planning  and  often  initial 
fabrication.  In  the  case  of  software,  the 
completion  of  the  CDR  initiates  the 
development  of  source  and  object  code.  The 
review  establishes  the  integrity  of  the  CSCI’s 
logical  design  prior  to  coding  and  testing. 
Typical  items  reviewed  at  the  CDR  include: 

a.  Draft  Type  C,  D,  and  E  specifications 
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b.  Detailed  engineering  drawings 

c.  Interface  control  drawings 

d.  Prototype  hardware 

e.  Manufacturing  Plan 

f.  Quality  Assurance  Plan. 

CDRs  are  the  last  major  design  reviews 
and  their  timing  controls  many 
pre-production  tasks  which  must  be  initiated 
to  prepare  for  the  transition  from  FSD  to 
Production.  The  correct  timing  of  CDRs  is 
critical.  A  balance  must  be  found  between 
technical  and  production  concerns.  For 
example,  closing  out  CDRs  early  can  provide 
more  time  for  production  but  could  reduce 
the  time  available  for  production  planning. 
In  other  words,  the  lack  of  full  maturity  of 
the  designs  reviewed  will  increase  the  risk 
that  the  number  of  post  review  changes  will 
adversely  affect  production  planning.  Closing 
out  CDRs  too  late  may  severely  constrain 
production  schedules  and  the  ability  to  obtain 
adequate  competition. 


12.2.6  Test  Readiness  Review 

The  TRR  is  a  formal  review  of  the 
contractor’s  readiness  to  begin  CSCI  testing. 
It  is  conducted  for  each  CSCI  after  informal 
testing  at  the  computer  software  component 
(CSC)  and  computer  software  unit  (CSU) 
level  for  that  CSCI  has  been  completed.  Its 
purpose  is  to  allow  the  government  to  1) 
determine  that  the  contractor  is  prepared  to 
begin  formal  testing  at  the  CSCI  level  in 
accordance  with  the  appropriate  software  test 


procedures,  2)  reach  a  technical 
understanding  on  the  validity  of  informal  test 
results,  and  3)  determine  the  adequacy  of  the 
software  operations  and  support  manuals. 
Items  reviewed  at  the  TRR  include: 

a.  Software  test  procedures  (STPR) 

b.  Draft  Computer  System  Operator’s 
Manual  (CSOM) 

c.  Draft  Software  User’s  Manual  (SUM) 

d.  Informal  CSC  and  CSU  test  and 
integration  results 

As  it  is  currently  defined,  the  TRR  does 
not  involve  any  specific  systems  engineering 
products.  However,  in  a  manner  similar  to 
all  other  reviews,  the  contractor  should 
demonstrate  that  the  functions  to  be  tested 
are  those  allocated  to  software  in  the 
underlying  technical  documentation,  and  that 
requirements  for  software  timing  and 
software  performance  will  support  the 
defined  mission  functional  requirements. 

12.2.7  Functional  Configuration  Audit 

The  objective  of  the  Functional 
Configuration  Audit  (FCA)  is  to  verify  that 
the  Cl’s  actual  performance  complies  with  its 
hardware/  software  development  and 
interface  requirements  specifications.  Test 
data  are  reviewed  to  verify  that  the  hardware 
and  software  performs  as  required  by  its 
functional/  allocated  configuration 
identification.  For  software,  a  technical 
understanding  is  reached  on  the  validity  and 
the  degree  of  completeness  of  the  Software 
Test  Reports  (STR),  and  as  appropriate, 
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updates  of  the  CSOM  and  SUM. 

The  FCA  for  a  complex  Cl  may  be 
conducted  on  a  progressive  basis  throughout 
the  Cl’s  development.  The  process  then 
culminates  after  the  completion  of  the 
qualification  testing  of  the  Cl  with  a  review 
of  all  Cl  discrepancies  at  the  final  system 
level  FCA.  The  FCA  must  be  conducted  on 
the  Cl  (whether  production,  prototype,  or 
pre-production)  representative  of  the 
configuration  to  be  released  for  production 
of  the  operational  inventory  quantities. 
When  a  prototype  or  pre-production  article 
is  not  produced,  the  FCA  will  be  conducted 
on  a  first  production  article.  For  cases  where 
Cl  qualification  can  only  be  determined 
through  integrated  system  testing,  FCAs  for 
such  CIs  will  not  be  considered  until 
integrated  system  testing  is  complete. 

122.8  Physical  Configuration  Audit 

The  Physical  Configuration  Audit  (PCA) 
is  the  formal  examination  of  the  as  built 
version  of  a  Cl  against  its  Type  C,  D,  and  E 
specification  documentation,  in  order  to 
establish  the  product  baseline.  After 
successful  completion  of  the  audit,  all 
subsequent  changes  are  processed  by 
engineering  change  action.  The  PCA  also 
determines  that  the  acceptance  testing 
requirements  prescribed  by  the 
documentation  are  adequate  for  acceptance 
of  production  units  of  a  Cl  by  quality 
assurance  activities. 

The  PCA  includes  a  detailed  audit  of 
engineering  drawings,  specifications,  technical 
data  and  tests  utilized  in  production  of 
hardware  CIs,  design  documentation,  code 
listings,  and  manuals  tor  CSCIs.  The  review 


also  includes  an  audit  of  the  released 
engineering  documentation  and  quality 
control  records  to  make  sure  the  as  built  or 
as  coded  configuration  is  reflected  by  this 
documentation.  For  software,  the  software 
product  specification  and  version  description 
document  (VDD)  shall  be  a  part  of  the  PCA 
review. 

In  complex  systems  containing  many  CIs, 
the  scope  of  an  audit  may  be  such  that  it 
must  be  broken  apart  to  ensure  effective 
participation  of  both  contractor  and 
government  personnel.  One  approach  used 
is  to  break  the  audit  into  three  stages: 

a.  Stage  1:  Review  of  Production  Baseline 

b.  Stage  2:  Operational  Audit 

c.  Stage  3:  Government  Acceptance  of 
the  Product  Baseline 

The  PCA  is  conducted  on  the  first  Cl 
production  article.  It  is  also  conducted  on 
those  items  that  are  a  reprocurement  of  a  Cl 
already  in  the  inventory.  A  PCA  must  be 
conducted  on  the  first  Cl  to  be  delivered  by 
a  new  contractor  even  though  a  PCA  was 
previously  accomplished  on  the  first  article 
delivered  by  a  different  contractor.  Formal 
approval  by  the  contracting  agency  of  the  Cl 
product  specification  (Type  C),  and  the 
satisfactory  completion  of  a  PCA,  results  in 
establishment  of  the  product  baseline  for  that 
Cl. 

122.9  Production  Readiness  Review 

The  objective  of  the  Production 
Readiness  Review  (PRR)  is  to  determine  if 
the  design  is  ready  for  production,  production 
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engineering  problems  have  been  resolved, 
and  adequate  planning  has  been 
accomplished  for  the  production  phase.  The 
PRR  represents  the  point  where  a  production 
commitment  can  be  made  without  incurring 
unacceptable  program  risk.  PRRs  should  be 
conducted,  by  the  systems  engineer,  as  a  time 
phased  effort  that  will  span  FSD  and 
encompass  the  developer/  producer  and 
major  subsystem  suppliers.  The  PRR 
examines  the  developer’s  design  from  the 
standpoint  of  completeness  and  producibility. 
It  examines  the  producer’s  production 
planning  documentation,  existing  and  planned 
facilities,  tooling  and  test  equipment, 
manufacturing  methods  and  controls,  material 
and  manpower  resources,  production 
engineering,  quality  control  and  assurance 
provisions,  production  management 
organization,  and  controls  over  major 
subcontractors.  The  result  of  the  PRR 
supports  the  systems  engineer’s  affirmative 
decision  at  the  production  decision  point  - 
that  the  system  is  ready  for  efficient  and 
economical  rate  production.  DoDI  5000.38 
[2]  identifies  the  requirements  and  criteria 
for  these  reviews.  Further  detail  on  the  PRR 
is  provided  in  Chapter  18  of  this  guide. 

12.2.10  Formal  Qualification  Review 

The  objective  of  the  Formal  Qualification 
Review  (FQR)  is  to  verify  that  the 
performance  of  the  CIs,  when  integrated  into 
a  system,  meets  all  system  functional 
requirements  and,  in  fact,  do  properly 
function  together  as  a  system.  The  FQR  will 
identify  the  test  report(s)/  data  that 
document  the  qualification  test  results  of  the 
CIs  to  show  compliance  with  the  system 
specification,  hardware  development 
specifications,  software  requirements 


specifications,  and  interface  requirements 
specifications.  Essentially,  the  FQR  is  a 
system  level  configuration  audit,  conducted 
after  full  up  system  testing  is  completed. 

When  required,  the  FQR  shall  be 
combined  with  the  FCA  at  the  end  of  Cl / 
subsystem  testing,  prior  to  PCA.  If  sufficient 
test  results  are  not  available  at  the  FCA  to 
ensure  that  CIs  will  perform  in  their  system 
environment,  or  deployed  operational 
experience  data  are  required,  the  FQR  will 
be  conducted  (post-  PCA)  during  follow  on 
operational  testing  and  evaluation  whenever 
the  necessary  tests  have  been  completed.  For 
non-combined  FCA s/  FQRs,  traceability, 
correlation,  and  completeness  of  the  FQR 
must  be  maintained  with  the  FCA  and 
duplication  of  effort  must  be  avoided.  For 
software,  a  technical  understanding  must  be 
reached  on  the  validity  and  degree  of 
completeness  of  the  Software  Test  Reports, 
and  as  appropriate,  updates  of  the  CSOM 
and  SUM  should  be  audited. 

The  point  of  government  certification  will 
be  determined  by  the  contracting  agency,  and 
will  depend  upon  the  nature  of  the  program, 
risk  aspects  of  the  particular  hardware  and 
software,  and  contractor  progress  in 
successfully  verifying  the  requirements  of  the 
CIs.  A  logical  time  to  schedule  an  FQR 
would  be  between  developmental  and 
operational  testing  of  the  system,  prior  to  the 
contractor  turning  the  system  over  to  the 
government. 

123  INFORMAL  REVIEWS 

Informal  reviews  may  include  technical 
interchange  meetings  called  either  by  the 
government  or  the  contractor  to  investigate 
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and  resolve  specific  technical  issues. 
Although  lower  level  reviews  are  held  on  an 
informal  basis,  they  still  require  an  agenda 
prepared  by  the  contractor  and  transmitted 
to  the  government  in  time  for  a  team  to  be 
assembled  and  briefed.  The  informal 
engineering  review  should  be  preceded  by  a 
data  package  from  the  contractor  containing 
applicable  engineering  drawings, 
specifications,  and  reports.  Meetings  must  be 
scheduled  to  ensure  that  required  technical 
experts  will  be  available.  The  use  of  material 
prepared  directly  by  engineers  should  be 
encouraged,  rather  than  formally  prepared 
material.  At  the  conclusion  of  the  review,  a 
summary  of  actions  and  approvals  should  be 
prepared  for  presentation  at  the  subsequent 
formal  review.  Issues,  questions,  agreements, 
and  action  items  must  be  documented  in 
minutes  and  assigned,  tracked,  and 
coordinated  with  appropriate  participants 
through  close  out  or  completion. 

Informal  engineering  reviews  are  not  the 
proper  fonun  for  considering  changes  to 
baselines  or  for  initiating  engineering  change 
proposals  (ECPs).  Review  of  changes  should 
be  reserved  for  Configuration  Control  Board 
(CCB)  meetings  following,  or  coinciding  with, 
formal  technical  review  meetings.  This 
assures  that  a  broad  representation  of 
management  and  technical  disciplines  will  be 
involved  in  the  review  and  approval  of 
engineering  change  proposals. 

12.4  TECHNICAL  REVIEW 
ADMINISTRATION 

In  developing  the  request  for  proposals 
(RFP)  for  any  acquisition  phase,  it  is 
necessary  for  the  government  to  identify 
specific  technical  reviews  to  be  imposed  on 
the  program.  This  is  generally  done  by 


tailoring  MIL-STD-1521B  [1]  to  fit  program 
needs.  MIL-STD-1521B  [1]  provides 
extensive  lists  of  topics  to  be  review’ed  for 
HWCIs  and  CSCIs;  support  equipment; 
electrical,  mechanical,  and  logical  designs; 
RAM;  and  many  other  design  characteristics. 
The  lists  should  be  used  to  develop  technical 
review  agendas  tailored  to  specific 
characteristics  of  the  program. 

The  RFP  will  also  identify  as  contract 
data  requirements  list  (CDRL)  items  the 
appropriate  technical  review  data  item 
descriptions  (DIDs).  These  include  technical 
review  agenda,  DI-A-3029  [3];  technical 
review  data  package,  DI-E-5423  [4];  and  the 
minutes  of  formal  reviews,  DI-E-3118  [5J. 
The  contractor  is  responsible  for  developing 
the  design  review  agenda  and  transmitting  it 
to  the  government  program  office  30  to  45 
days  before  the  meeting.  The  systems 
engineer  notes  any  changes  in  content  and 
approves  the  final  agenda.  The  contractor 
prepares  the  technical  review  data  package 
and  transmits  it  to  the  government,  as 
specified  in  the  CDRL,  two  or  more  weeks 
before  the  meeting.  Following  review  of  the 
data  package,  questions  from  the  team  are 
coordinated  by  the  systems  engineer  and 
transmitted  to  the  contractor  before  the 
meeting,  as  contractually  specified. 

The  success  of  a  review  depends  on  both 
government  and  contractor  preparations 
before  the  meeting.  It  is  important  that  the 
technical  review  teams  be  composed  of 
personnel  competent  to  cover  all  areas  to  be 
reviewed  and  that  the  government  members 
have  sufficient  time  available  to  review  the 
planned  agenda  and  data  package  before  the 
meeting.  Technical  reviews  should  l 't  be 
tutorial  sessions.  Personnel  training  is  not 


12-11 


the  objective  of  technical  reviews.  All 
government  participants  should  meet  before 
the  scheduled  review  to  acquaint  themselves 
with  material  being  reviewed,  objectives  of 
the  review,  and  the  approach  to  be  used  in 
the  review.  Review  personnel/  teams  should 
be  specifically  chartered  by  the  systems 
engineer  to  cover  specific  areas,  documents, 
or  subjects.  It  is  essential  that  the  teams  be 
held  to  the  minimum  number  of  personnel 
to  adequately  cover  the  areas  of  the  review 
so  that  excessive  time  is  not  spent  on 
discussion.  A  list  of  authorized  government 
attendees  is  provided  to  the  contractor  prior 
to  the  meeting  to  control  participation, 
ensure  qualified  personnel,  and  observe 
appropriate  security  regulations. 

The  government  and  contractor  systems 
engineers  act  as  co-chairmen  of  the  technical 
review.  The  government  ensures  that  all 
areas  are  adequately  examined  and  provides 
coordinated  comments  and  direction  to  the 
contractor.  Only  the  designated  co-chairmen 
have  decision  making  responsibilities,  so  they 
must  be  thoroughly  familiar  with  all  aspects 
of  the  program  before  the  review  convenes. 
A  concerted  effort  must  be  made  to  avoid 
an  adversarial  relationship,  which  sometimes 
results  from  pressures  of  program  schedule 
and  funding  limits.  Candor  and  objectivity 
should  be  encouraged  among  all  participants. 
Teamwork  should  be  stressed.  The  objective 
of  the  technical  review  is  to  search  out 
design  weaknesses  or  faulty  designs.  This 
can  be  accomplished  through  healthy 
skepticism,  not  through  an  adversary  role.  A 
properly  selected/  prepared  review  team  can 
provide  invaluable  assistance  to  the 
contractor’s  systems  engineering/  design 
teams  in  refining  trade-offs  and  synthesizing 
design  or  production  alternatives.  Problems 


should  be  identified  early  from  the  data 
package  submittal  and  attempts  should  be 
made  to  clarify  them  on  an  individual  basis 
before  the  review.  Avoid  presenting  major 
problem  surprises  at  the  meeting.  This 
saves  meeting  time  and  helps  maintain 
positive  attitudes. 

Decisions,  agreements,  and  approved 
action  items  should  be  recorded  and  signed 
by  both  the  government  and  the  contractor 
at  the  end  of  each  day.  Each  action  item 
should  be  assigned  to  responsible  individuals 
(one  government,  one  contractor),  with  an 
assigned  closure  date  and  the  type  of 
response  required  to  close  the  action.  The 
co-chairmen  should  evaluate  proposed 
actions  to  ensure  that  they  are  not  out  of 
contract  scope.  Out  of  scope  changes  must 
be  handled  by  contracting  officers  in 
accordance  with  applicable  procurement 
regulations  and  CCB  procedures. 

After  each  formal  technical  review,  the 
minutes  of  that  review  are  published  for 
distribution.  The  minutes  identify  the 
proceedings  of  the  review,  as  well  as  action 
items,  for  accomplishment/  resolution.  The 
government  provides  formal 
acknowledgement  to  the  contractor  of  the 
accomplishment  of  each  review  after  receipt 
of  the  review  minutes  and  satisfactory  close 
out  of  all  outstanding  action  items.  The 
government  establishes  the  adequacy  of  the 
contractor’s  review  performance  by 
notification  of  approval,  contingent  approval 
(review  will  be  accomplished  upon 
satisfactory  completion  of  the  action  items), 
or  disapproval  (indicating  that  the  review 
was  seriously  inadequate).  It  should  be 
noted  that  formal  technical  review 
completion  is  a  good  indicator  of  overall 
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program  status.  Slippage  in  review 
schedules  will  often  lead  to  slippage  in 
overall  program  plans  including  scheduling  of 
major  acquisition  milestones. 

12.4.1  Technical  Review  Documentation 

The  technical  review  data  package 
consists  of  documentation  developed  by  the 
contractor  according  to  DID  DI-E-5423  [4j 
during  the  development  process,  and  may 
include  the  following: 

a.  Specifications 

b.  Configuration  and  lay-out  drawings 

c.  Analysis  and  simulation  reports 

d.  Trade  study  reports 

e.  FFBDs,  RASs,  TLSs,  SBDs,  DSs 

f.  Plans 

g.  RAM  data 

h.  Survivability/  Vulnerability  (S/V)  data 

i.  Verification  data. 

If  more  data  are  requested  than  can  or 
should  be  reviewed  by  the  systems  engineer 
in  the  time  available  before  the  review,  an 
unnecessary  cost  is  incurred  and  review 
credibility  is  lost.  Coordination  with  the 
contractor  is  therefore  required  to  ensure 
the  efficiency  of  providing  only  needed  data. 

12.4.2  Technical  Review  Meeting  Minutes 

Technical  review  meeting  minutes  are 


prepared  by  the  contractor  according  to  DID 
DI-E-3118  [5]  and  should  include  the 
following: 

a.  Narrative  summary  of  significant 
comments,  findings,  decisions,  and  direction 
provided  at  the  meeting,  with  rationale 
where  appropriate 

b.  Meeting  agenda 

c.  List  of  data  package  contents 

d.  List  of  attendees 

e.  Action  items  with  responsibilities  and 
due  dates 

f.  List  of  presentation  material. 

12.5  RELATION  OF  TECHNICAL 
REVIEWS  TO  PROGRAM 
SPECIFICATIONS 

Program  specifications  are  the  basic  tools 
for  establishing  the  formal  baselines  during 
the  development  cycle.  Figure  12-2,  shown 
previously,  summarizes  the  preceding 
technical  reviews,  related  general  schedules, 
and  specifications  for  hardware  and  software 
development  that  are  primary  candidates  for 
review.  Generally  speaking  1)  the  system 
functional  baseline  is  normally  established  at 
the  SDR  and  is  represented  by  tne  approved 
system  specification  (Type  A);  2)  the 
allocated  baselines  for  HWCIs  are  normally 
established  at  their  PDRs,  but  no  later  than 
their  CDRs  and  are  represented  by  their 
approved  development  specifications  (Type 
B);  3)  the  allocated  baselines  for  CSCIs  are 
normally  established  at  their  SSRs  and  are 
represented  by  their  approved 
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developmental  specifications  (software 
requirements  and  interface  requirments 
specifications;  Type  B-5);  and  4)  the  product 
baselines  for  HWCIs  and  CSCIs  are 
normally  established  at  their  PCAs  and  are 
represented  by  their  product  specifications 
(Type  C).  For  greater  detail  on  the 
development  of  program  specifications,  see 
Chapter  10  of  this  guide. 

12.6  RISK  TEMPLATE:  TECHNICAL 
REVIEWS 

The  application  of  technical  review 
requirements  involves  the  balancing  of  many 
factors.  The  timing  of  reviews,  scope  of 
review,  government  and  contractor  personnel 
who  participate,  and  preparation  of 
personnel  are  all  factors  that  may  lead  to 
successful  or  unsuccessful  implementation. 
While  defense  contracts  usually  require 
formal  technical  reviews,  they  often  lack 
specific  direction  and  discipline  in  the 
technical  review  requirement,  resulting  in  an 
unstructured  review  process  that  fails  to 
fulfill  either  of  the  two  main  purposes  of 
technical  review,  which  are:  (1)  to  bring 
additional  knowledge  to  the  design  process 
to  augment  the  basic  program  design  and 
analytical  activity;  and  (2)  to  challenge  the 
satisfactory  accomplishment  of  specified 
design  and  analytical  tasks  needed  for 
approval  to  proceed  with  the  next  step  in 
the  acquisition  process.  DoD  4245.7-M  [6] 
recommends  the  following  measures  for 
reducing  risk: 

a.  Government  and  contractors  should 
recognize  that  technical  reviews  represent 
the  "front  line"  for  determining  program 
readiness  to  transition  to  the  next 
development  phase.  Technical  review  policy, 


schedule,  budget,  agenda,  participants, 
actions,  and  follow-up  should  be  decided  in 
view  of  this  need. 

b.  A  technical  review  plan  outlining  the 
schedule  of  reviews  should  be  developed  by 
the  contractor  and  approved  by  the 
government.  The  technical  review  plan 
should  include  both  government  and  internal 
contractor  technical  reviews  and  inspections. 

c.  Techncial  review  requirements  should 
be  allocated  to  subcontractors  and  suppliers 
to  ensure  proper  subcontractor  internal 
technical  review  practices  and  to  provide 
timely  opportunities  for  both  the  contractor 
and  government  to  challenge  subcontracted 
material  design. 

d.  Government  and  contractor  techncial 
review  participants  should  be  selected  or 
recruited  from  outside  the  program  to  be 
reviewed,  on  the  basis  of  experience  and 
expertise  in  challenging  the  design.  They 
should  have  a  collective  technical 
competence  greater  than  or  equal  to  that  of 
the  designers  responsible  for  the  design 
under  review. 

e.  Manufacturing,  product  assurance,  and 
logistics  engineering  functions  should  be 
represented  and  have  authority  equal  to 
engineering  in  challenging  design  maturity. 

f.  Technical  reviews  should  use  computer- 
aided  design  analyses,  whenever  available, 
and  include  review  of  production  tooling 
required  at  the  specific  program  milestone. 

12.7  SUMMARY 

A  summary  of  major  areas  of 
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Major  Technical  Review  Concerns 
Figure  12-3 


Timing/Schedul ing 

- 

Do  design  reviews  support  program  milestone  decisions? 

- 

Do  technical  reviews  and  audit  schedules  consider  the 
availability  of  appropriate  program  documentation, 
hardware,  software,  and  test  results? 

- 

Are  design  reviews  scheduled  late  enough  to  ensure 
adequate  design  definition? 

- 

Are  design  reviews  scheduled  early  enough  to  avoid 
costly  corrections  to  the  design? 

Tai lor ing/Scope 

- 

Have  the  design  review  requirements  of  MIL-STD-1521B 
been  tailored  to  program  needs? 

- 

Have  data  requirements  supporting  reviews  been 
tailored  so  that  only  necessary  data  are  produced 
before,  during,  and  after  the  review? 

_ 

Are  the  right  data  available  to  make  decisions 
required  by  the  review? 

- 

Is  the  scope  of  the  review  appropriate  to  the  stage  of 
design  maturity? 

- 

Is  the  scope  of  the  review  adequate  to  accomplish 
review  objectives? 

Personne 1 

- 

Are  all  appropriate  government  and  contractor 
personnel  scheduled  to  participate  in  each  review? 

- 

Are  all  personnel  familiar  with  the  topics  to  be 
covered,  and  are  they  prepared  for  the  review? 

- 

Is  the  review  team  small  enough  to  focus  the 
discussion? 

Administration 

- 

Does  the  action  item  list  prepared  during  the  review 
assign  each  action  to  an  individual  responsible  for 
its  resolution  within  a  specified  time  period? 

- 

Is  the  effort  to  close  out  the  review  reasonable? 
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management  concern  is  shown  in  Figure 
12-3.  There  are  no  easy  guidelines  to 
achieving  an  optimum  balance  among  these 
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CHAPTER  13 


ROLE  OF  TEST  AND  EVALUATION  IN 
THE  SYSTEMS  ENGINEERING  PROCESS 


13.1  INTRODUCTION 

In  the  early  1970s,  Department  of 
Defense  (DoD)  test  policy  became  more 
formalized  and  placed  greater  emphasis  on 
test  and  evaluation  (T&E)  as  a  continuing 
function  throughout  the  acquisition  cycle. 
These  policies  stressed  the  use  of  T&E  to 
reduce  acquisition  risk  and  provide  early 
and  continuing  estimates  of  the  system’s 
operational  effectiveness  and  operational 
suitability.  In  order  to  meet  these 
objectives,  it  is  necessary  that  appropriate 
test  activities  be  fully  integrated  into  the 
overall  development  process.  From  a 
systems  engineering  perspective,  test 
planning,  testing,  and  analysis  of  test 
results  are  integral  parts  of  the  basic 
systems  engineering  process. 

The  integration  of  T&E  requirements 
has  several  dimensions  which  includes  two 
broad  categories  of  testing:  1)  government 
and  2)  contractor.  Government  tests  can 
be  further  categorized  as  user  tests,  which 
are  broadly  operational  in  emphasis,  and 
builder  tests,  which  focus  on  achievement 
of  development  requirements. 

Another  perspective  of  T&E 
requirements  shows  that  test  and 
evaluation  encompasses  relationships  with 
all  system  elements:  equipment,  software, 
facilities,  personnel,  and  procedural  data. 


Each  work  breakdown  structure  (WBS) 
element  must  receive  appropriate  T&E.  In 
most  cases  (e.g.,  software)  the  system  element 
may  have  unique  requirements  which  constrain 
the  approach  taken  to  testing. 

Another  T&E  dimension  to  consider  is 
that  testing  spans  the  overall  acquisition  life 
cycle.  It  is  not  simply  something  that  takes 
place  when  development  is  complete.  Finally, 
as  T&E  requirements  are  identified  for  the 
operation  (and  support)  functions,  the  systems 
engineering  process  can  also  be  used  to 
identify  the  system  elements  (resources  and 
procedures)  necessary  for  the  test  activities 
themselves. 

T&E  policy,  described  in  DoDD  5000.3 
[I],  provides  guidelines  for  planning  and 
conducting  test  and  evaluation.  It  defines  and 
describes  the  major  categories  of  Development 
Test  and  Evaluation  (DT&E)  and  Operational 
Test  and  Evaluation  (OT&E),  and  provides 
for  exceptions  such  as  combining  DT&E  with 
OT&E,  T&E  for  special  acquisition  programs, 
T&E  of  computer  software,  T&E  of  system 
alterations,  and  joint  T&E  programs.  DoDD 
5000.3  [1]  specifies  three  general 

requirements: 

a.  Successful  accomplishment  of  T&E 
objectives  will  be  a  key  requirement  for 
decisions  to  commit  significant  additional 
resources  to  a  program  or  to  advance  it  from 
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one  acquisition  phase  to  another. 

b.  T&E  shall  begin  as  early  as 
possible  and  be  conducted  throughout  the 
system  acquisition  process  to  assess  and 
reduce  acquisition  risks,  and  to  estimate 
the  operational  effectiveness  and 
operational  suitability  of  the  system. 

c.  The  dependence  on  subjective 
judgement  of  system  performance  will  be 
minimized  dunr-g  testing. 

The  dire  .uv.j  responsibilities  of 

the  ‘  .mny  j.hreo  cv  Defense  Research 
and  Lag:.  cering  (Test  and  Evaluation) 
DDDRE(T&E)  and  the  Director, 
Operational  Test  and  Evaluation 
(DOT&E).  The  general  organization  for 
DoD  T&E  is  illustrated  in  Figure  13-1. 


DoDD  5000.3  [1]  is  supported  by  five 
manuals  (see  References  [2]  through  [5])  in 
the  areas  of: 

a.  Test  and  Evaluation  Master  Plans 
(TEMPs). 

b.  DoD  NATO  comparative  tests  and 
foreign  weapons  evaluation. 

c.  Software  T&E. 

d.  Joint  T&E  procedures. 

e.  DT&E  support  of  major  weapon  systems 
operational  testing. 

In  summary,  there  is  clear  policy  stating 
test  and  evaluation  program  requirements, 
with  particular  emphasis  on  those  programs 


OoD  T&E  Organizational  Struoturo 
Figure  1S-1 
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designated  as  major  weapon  systems.  Test 
and  evaluation  is  an  integral  part  of  the 
systems  engineering  management  process. 
It  begins  early  and  extends  throughout  the 
acquisition  life  cycle.  The  most  general 
objectives  of  the  Test  and  Evaluation 
program  are  1)  to  assess  and  reduce  the 
risk  to  the  program  and  2)  to  estimate  the 
operational  suitability  and  effectiveness  of 
the  system. 

13.2  TESTING 

Test  and  evaluation  (T&E)  is 
conducted  throughout  the  system 
acquisition  life  cycle.  There  are  two  major 
categories  of  T&E:  1)  Development  Test 
and  Evaluation  (DT&E)  and  2) 
Operational  Test  and  Evaluation  (OT&E). 
Figure  13-2  shows  how  DT&E  and  OT&E 
are  integrated  into  the  acquisition  life 
cycle. 

13.2.1  Developmental  Test  And 
Evaluation 

Development  test  and  evaluation 
(DT&E)  is  conducted  throughout  the 
acquisition  process  to  ensure  the 
acquisition  and  fielding  of  an  effective  and 
supportable  system.  DT&E  includes  test 
and  evaluation  of  components  and 
subsystems  at  all  work  breakdown  structure 
(WBS)  levels  including  preplanned  product 
improvement  (P^I)  changes,  hardware/ 
software  integration,  and  related  software, 
as  well  as  qualification,  live  fire,  and 
production  acceptance  testing.  It  involves 
the  use  of  simulations,  models, 
breadboards,  brassboards,  and  testbeds,  as 
well  as  full  scale  engineering  development 
models  or  prototypes  of  system 


components  or  the  system  itself. 

DT&E  is  normally  planned,  conducted,  and 
monitored  by  the  developing  agency.  DT&E 
is  conducted  to: 

a.  Assist  the  engineering  design  and 
development  process 

b.  Verify  performance  objectives  and 
specifications 

c.  Demonstrate  that  design  risks  have  been 
minimized 

d.  Estimate  the  system’s  utility  when 

introduced 

e.  Evaluate  the  compatibility  and 
interoperability  with  existing  or  planned 
equipment/  systems 

f.  Provide  an  assurance  that  the  system/ 
equipment  is  ready  for  testing  in  the 
operational  environment. 

DT&E  is  divided  by  acquisition  phase.  It 
is  often  abbreviated  "DT".  Since  each  phase 
of  DT  supports  the  next  milestone  decision,  it 
must  be  completed  and  documented  far 
enough  in  advance  so  that  the  results  can  be 
used  in  reaching  that  decision.  End-of-phase 
test  reports  for  major  defense  acquisition 
programs  must  be  provided  to  the  DOT&E 
and  DDRE(T&E)  least  45  days  prior  to  a 
milestone  decision  or  the  final  decision  to 
proceed  beyond  low-rate  initial  production 
(LRIP). 

DT  is  conducted  1)  during  the  Concept 
Exploration/  Definition  (C/E)  phase  to  assist 
in  selecting  preferred  alternative  system 
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Figure  13-2 

Testing  in  the  Acquisition  Process 


13-4 


ADV  PROTOTYPE 


concepts,  technologies,  and  designs;  2) 
during  the  Concept  Demonstration/ 
Validation  (D/V)  phase  to  identify  and 
validate  the  preferred  technical  approach, 
including  the  identification  of  technical 
risks  and  feasible  solutions;  3)  during  the 
Full  Scale  Development  (FSD)  phase  to 
demonstrate  that  engineering  is  reasonably 
complete,  that  all  significant  design 
problems  have  been  identified  and 
solutions  to  these  problems  are  in  hand, 
and  that  the  design  meets  its  required 
specifications  in  all  areas  (such  as 
performance,  reliability,  and 
maintainability)  within  the  range  of 
environmental  parameters  designed  for  the 
operational  employment  of  the  system;  and 
4)  after  the  Milestone  III  (Production  and 
Deployment)  Decision.  It  is  an  integral 
part  of  the  development,  validation,  and 
introduction  of  system  changes  undertaken 
to  improve  the  system,  to  react  to  new 
threats,  and/  or  to  reduce  life  cycle  costs. 

As  part  of  DT,  each  developing  agency 
is  also  responsible  for  tl.a  qualification 
testing  that  verifies  the  design  and  the 
manufacturing  process  and  provides  a 
baseline  for  subsequent  acceptance  tests. 
Qualification  tests  consist  of  pre-production 
and  production  qualification  tests. 

Pre-production  qualification  tests  are 
formal  contractual  tests  that  ensure  design 
integrity  over  the  specified  operational  and 
environmental  range.  These  tests  usually 
use  pre-production  or  prototype  hardware 
fabricated  to  the  proposed  production 
specifications  and  drawings.  Such  tests 
include  reliability  and  maintainability 
demonstration  tests  required  prior  to 
production  release.  Pre-production 


qualification  testing  must  be  completed  prior 
to  Milestone  III  as  it  is  a  critical  factor  in 
assessing  the  system’s  readiness  for  production. 

Production  qualification  tests  are 
conducted  for  all  production  items  to  ensure 
the  effectiveness  of  the  manufacturing  process, 
equipment,  and  procedures.  All  new 
production  items  are  subjected  to  first  article 
test  to  verify  specification  compliance  and 
form,  fit  and  function.  Production  acceptance 
tests  are  conducted  on  each  item  or  on  a 
sample  lot  taken  at  random  from  each 
production  lot.  These  tests  are  repeated  when 
the  process  or  design  is  changed  significantly, 
and  when  a  second  or  alternate  source  is 
brought  on  line.  Production  qualification  tests 
are  also  conducted  against  contractual 
requirements. 

13 22  Operational  Test  and 
Evaluation 

For  major  systems,  OT&E  is  normally 
planned  and  conducted  by  a  major  OT&E 
field  agency  located  within  the  DoD 
component.  This  Operational  Test  Agency 
(OTA)  must  be  separate  and  independent 
from  both  the  developing/  procuring  agency 
and  the  using  agency.  The  OTA  is 
responsible  for  managing  operational  testing, 
reporting  test  results,  and  providing  directly  to 
the  Military  Service  Chief  or  Defense  Agency 
Director  its  independent  evaluation  of  the 
system  being  tested.  The  principal  objectives 
of  OT&E  are  to: 

a.  Estimate  the  operational  effectiveness  and 
operational  suitability  of  the  system. 

b.  Identify  needed  modifications  or 
improvements. 
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c.  Provide  the  information  on  tactics, 
doctrine,  organization,  and  personnel 
requirements. 

d.  Provide  data  to  uphold  or  verify  the 
adequacy  of  various  manuals,  handbooks, 
supporting  plans,  and  documentation. 

Although  OT&E  is  planned  and 
conducted  by  an  independent  testing 
activity,  the  Program  Manager  (PM)  must 
closely  coordinate  all  aspects  of  test  and 
evaluation  with  this  organization  to  ensure 
that  DT&E  objectives  coincide  with  OT&E 
objectives. 

OT&E  is  conducted  in  an  environment 
as  operationally  realistic  as  possible. 
Typical  operation  and  support  personnel 
will  be  used  to  obtain  a  valid  estimate  of 
the  user’s  capability  to  operate  and 
maintain  the  system  when  deployed  under 
both  peacetime  and  wartime  conditions. 
The  system  contractor  is  precluded  by 
public  law  99-661  from  participating  in 
realistic  operational  test  and  evaluation. 
OT&E  is  often  abbreviated  "OT".  Each 
phase  of  OT,  like  DT,  supports  the  next 
milestone  decision. 

OT  is  conducted  1)  during  C/E  to 
estimate  the  operational  impact  of 
candidate  technical  approaches  and  to 
assist  in  selecting  preferred  alternative 
system  concepts.  Most  acquisition 
programs  will  require  early  operational 
assessments  (EOA);  2)  during  D/V  to 
examine  the  operational  aspects  of  the 
selected  alternative  technical  approaches, 
estimate  the  potential  operational 
effectiveness  and  suitability  of  the 
candidate  system,  and  identify  operational 


issues  for  the  early  operational  assessment  and 
FSD  OT;  3)  during  FSD  to  demonstrate  the 
system’s  operational  effectiveness  and 
suitability.  The  items  tested  must  sufficiently 
represent  expected  production  models  to 
ensure  that  a  valid  assessment  of  the  system 
can  be  made.  The  development  of  tactics  for 
operational  employment  of  the  system  within 
the  range  of  environmental  parameters  should 
be  well  underway;  and  (4)  after  Milestone  III, 
but  before  the  production  system  is  ready  for 
testing.  Normally,  limited  follow-on  OT  will 
use  the  same  system  and  support  equipment 
used  in  the  operational  evaluation  and  will  test 
the  fixes  to  be  incorporated  in  production 
systems,  complete  deferred  or  incomplete 
pre-production  test  and  evaluation,  and 
continue  tactics  development.  FOT&E  will 
continue  until  the  objectives  specified  in  the 
approved  TEMP  for  this  phase  have  been 
met,  regardless  of  the  date  of  deployment  of 
production  systems. 

FOT&E  is  conducted  on  production 
systems  to  validate  the  achievement  of  the 
program  objectives  for  production  system 
operational  effectiveness  and  suitability.  Other 
OT  testing  may  include  OT  of  the  existing 
system  in  a  new  environment,  with  a  new 
subsystem,  in  a  new  tactical  application,  or 
against  a  new  threat. 

Since  DT  and  OT  take  place  during  the 
same  phases  of  the  acquisition  cycle,  it  may 
make  sense  to  coordinate  early  DT  and  OT 
testing  to  use  resources  more  efficiently  in 
obtaining  the  data  necessary  to  satisfy  the 
common  needs  of  both  the  developing  agency 
and  OTA.  This  is  called  combined  testing. 
Development  and  operational  tests  can  be 
combined  when  significant,  clearly  identified 
cost  and  time  benefits  will  result.  Of  course, 
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the  test  objectives  of  both  the  developing 
agency  and  OTA  will  have  to  be  reflected 
in  this  combined  testing  situation. 

One  important  note:  The  FSD  subphases 
of  DT  technical  evaluation  and  OT 
operational  evaluation  should  not  be 
combined. 

13J  APPROACH 

Although  all  acquisition  programs  are 
tailored  to  system  requirements,  they 
generally  follow  a  sequence  similar  to  that 
illustrated  in  Figure  13-3.  The  C/E  phase 


considers  various  ways  of  satisfying  the 
operational  requirement  and  identifies  the 
preferred  alternative  or  alternatives.  Often  it 
includes  test  requirements  related  to  advanced 
technology. 

The  D/V  phase  results  in  the  fabrication 
and  test  of  functional  hardware.  Test  and 
evaluation  during  this  phase  validates  the 
approach  and  demonstrates  that  equipment  or 
systems  can  be  built  to  meet  the  operational 
requirements. 

FSD  produces  the  full  scale  engineering 
development  or  service  test  models  that 
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closely  approximate  the  expected 
production  articles.  These  models  are 
used  for  rigorous  test  and  evaluation  in  an 
operational  environment  and  for 
assessment  of  the  supportability  of  the 
system.  The  tests  should  demonstrate  that 
the  proposed  production  system  will  meet 
its  operational  requirements  and  can  be 
operated,  maintained,  and  fully  supported 
while  in  operational  use. 

In  the  Production  phase,  production  of 
the  equipment  in  appropriate  quantities 
and  deployment  to  field  units  with 
supporting  equipment,  repair  parts,  and 
documentation  lead  to  the  initial  operating 
capability  (IOC)  to  complete  the  formal 
development  process  and  provide  the  final 
testing  environment. 

A  major  controlling  document  for  every 
acquisition  program  is  the  TEMP,  which 
lays  out  the  overall  schedule  of 
development  and  operational  test  and 
evaluation.  This  test  and  evaluation 
process  will  verify  that  the  new  equipment 
does,  in  fact,  meet  the  requirements.  The 
specific  content  of  the  TEMP  is  described 
in  greater  detail  in  Section  13.9  and  in 
DoD  5000.3-M-l  (2]. 

The  T&E  program  is  in  some  ways 
similar  to  the  basic  systems  engineering 
process.  Figure  13-4  illustrates  some  key 
relationships  between  the  systems 
engineering  process  and  T&E  activities. 

133.1  Concept  Exploration/  Definition 
Phase  Activity 

Although  the  TEMP  is  not  required  in 
final  form  until  the  end  of  the  C/E  phase, 


draft  sections  of  the  TEMP  may  be  prepared 
during  C/E.  As  systems  engineering 
information  becomes  available  you  can  begin 
preparing  Part  I,  Program  Background,  of  the 
TEMP.  Functional  flow  block  diagrams 
(FFBDs),  requirements  allocation  sheets 
(RASs),  and  test  requirements  sheets  (TRSs) 
provide  the  principal  supporting  systems 
engineering  documentation.  During  this 
phase,  systems  engineering  documentation 
used  in  the  trade  study  process  provides  the 
principal  evaluation  tools  for  selection  among 
alternative  concepts.  If  testing  does  take 
place,  it  is  not  likely  to  be  on  a  prototype  of 
the  system,  but  more  commonly  on  an 
experimental  component  that  may  be  the 
heart  of  the  development  effort.  Testing  is 
limited  in  order  to  devote  no  more  resources 
than  necessary  to  tne  process  of  selecting  a 
concept.  Sometimes  C/E  phase  testing 
consists  of  adapting  a  currently  fielded  system 
to  investigate  its  potential  for  another 
application.  If  test  reports  are  developed, 
they  are  usually  done  by  a  technical  agent  or 
laboratory. 

Some  smaller  programs  have  a  less  formal 
C/E  phase  where  limited  testing  does  take 
place.  Sometimes  the  objective  is  not  to 
select  a  concept,  but  more  simply  to 
demonstrate  the  feasibility  of  a  known  concept 
or  technology  alternative.  This  usually  takes 
the  form  of  "black  box"  testing  at  a  laboratory, 
when  a  single  performance  attribute  is  in 
question. 

At  Milestone  I,  the  PM  should  address 
alternative  design  concepts,  alternative 
acquisition  strategies,  expected  operational 
capabilities,  industrial  base  capacity,  readiness, 
support,  personnel  requirement  projections, 
and  cost  estimates. 


13-8 


13-9 


T&E  planning  at  Milestone  I  should 
include  an  outline  of  the  program  and 
performance  thresholds  that  are  to  be 
demonstrated  during  the  D/V  phase. 
Systems  engineering  documentation,  such 
as  the  RAS  and  TRS,  is  the  primary 
source  for  these  thresholds.  Thresholds 
are  normally  monitored  through  the 
technical  performance  measurement 
program  (see  Chapter  14  of  this  guide). 
Programs  are  required  by  DoDD  5000.3 
[1]  to  have  a  TEMP  at  Milestone  I. 

For  those  programs  where  a  Milestone 
I  decision  is  not  required,  a  TEMP  is  still 
required  around  the  time  that  the 
milestone  would  have  occurred  --  the 
beginning  of  the  first  fiscal  year  containing 
program  funding.  For  programs  requiring 
a  TEMP  at  Milestone  I,  the  TEMP  must 
be  prepared  during  the  C/E  phase.  At 
the  latest,  it  should  be  developed,  staffed, 
and  approved  prior  to  the  conduct  of  any 
major  OT&E  events.  Since  the  negotiation 
and  approval  process  for  TEMPs  can  be 
time  consuming,  it  is  strongly 
recommended  that  the  Program  Manager 
begin  preparing  the  TEMP  as  early  in  the 
program  as  possible  using  the  systems 
engineering  documentation  as  a  source  of 
requirements.  It  is  recognized  that  the 
initial  versions  of  the  TEMP  will  lack 
specifics,  particularly  in  regard  to  test 
planning  more  than  two  years  away. 
Nevertheless,  this  should  not  discourage 
early  preparation  using  the  level  of  systems 
engineering  documentation  available. 

13 3.2  Concept  Demonstration/  Validation 
Phase  Activity 

In  the  D/V  phase,  concepts  that  were 


generated  during  the  C/E  phase  and  selected 
for  continuation  are  verified.  This  verification 
usually  includes  some  hardware  fabrication 
and  demonstration.  Program  risks  and 
uncertainties  are  identified,  and  some  are 
resolved.  Also  during  this  phase,  the  system’s 
allocated  baseline  configuration  and  other 
documentation  necessary  to  enter  the  FSD 
phase  are  prepared  based  on  the  systems 
engineering  process  and  documentation. 

The  scope  of  each  program’s  D/V  effort 
is  primarily  driven  by  the  extent  of  new 
engineering  development  associated  with  the 
design.  Generally,  programs  that  use  current 
technology  have  short  D /V  phases.  The  C/E 
and  D/V  phases  can  even  be  combined  for 
such  programs.  In  others,  the  design, 
fabrication,  and  testing  of  one  or  more 
advanced  development  models  (ADMs)  will  be 
necessary. 

The  objectives  of  testing  during  the  D/V 
phase  are:  1)  to  verify  that  the  areas  of 
technical  risk  to  be  resolved  during  that  phase 
are  evaluated;  2)  to  assist  in  identifying  the 
preferred  technical  approach  to  be  taken 
during  FSD;  3)  to  verify  that  the  system  has, 
at  a  minimum,  the  potential  to  be  technically 
and  operationally  effective,  as  well  as 
operationally  suitable;  and  4)  to  determine  the 
T&E  requirements  to  be  achieved  during 
FSD.  To  accomplish  this,  adequate  DT  must 
be  conducted  on  each  alternate  system.  If  it 
is  expected  that  the  system  will  employ  new 
operational  concepts  or  involve  significant 
operational  risks,  OT  or  an  EOA  may  be 
conducted  on  the  more  promising  systems, 
that  is,  those  that  meet  most  or  all  of  the 
performance  objectives.  Sufficient  OT  is 
conducted  to  assess  the  operational 
effectiveness  and  suitability  of  each 
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recommended  alternative. 

DT  and  OT  conducted  on  the 
candidate  system  components/  subsystems 
may  evaluate  such  factors  as  producibility, 
compatibility,  reliability,  interoperability, 
electromagnetic  environmental  effects, 
logistics  supportability,  and  maintainability. 
Normally  these  requirements  will  be 
identified  using  the  RAS  and  TRS  as 
primary  systems  engineering 
documentation.  The  preferred  approach  is 
selected  through  trade  studies  involving 
T&E  results  and  other  factors  such  as 
development,  production,  and  life  cycle 
costs. 

At  the  FSD  decision,  Milestone  II,  the 
PM  should  be  prepared  to  answer  these 
T&E  related  questions: 

a.  Have  the  technical  questions  and 
critical  issues  posed  at  Milestone  T  been 
adequately  resolved? 

b.  Has  the  T&E  conducted  been 
sufficient  to  reasonably  ensure  that  the 
performance  requirements  can  be 
achieved? 

c.  Have  the  technical  questions  and 
critical  issues  to  be  resolved  prior  to 
Milestone  III  been  identified?  Is  the  test 
program  to  resolve  them  adequately 
planned? 

d.  Have  the  technical  and  operational 
performance  requirements  and  thresholds 
been  refined?  Have  the  provisions  for 
demonstration  of  these  requirements  and 
thresholds  been  included  in  the  updated 
TEMP? 


1333  Full  Scale  Development  Phase 
Activity 

The  FSD  phase  encompasses  the  final 
engineering,  prototyping,  and  testing  necessary 
to  demonstrate  and  document  that  the  system 
is  cost-effective,  operationally  effective,  and 
operationally  suitable.  The  final  product  is  a 
baseline  design  for  use  during  production. 

For  system  acquisition  programs  with 
moderate  to  high  technical  risks,  FSD  often 
involves  the  fabrication  and  test  of  one  or 
more  engineering  development  models 
(EDMs)  followed  by  the  fabrication  and  test 
of  one  or  more  LRIP  models.  For  programs 
of  lesser  risk,  it  may  involve  pilot  production 
models  only. 

When  the  design  is  validated  sufficiently, 
pre-production  models  are  manufactured  to 
approximate  the  final  intended  production 
configuration  as  closely  as  possible.  Usually, 
the  number  of  units  is  limited  to  the  amount 
needed  to  conduct  FSD  testing.  Occasionally, 
a  larger  number  may  be  authorized  for  a 
variety  of  reasons,  such  as  the  urgent  need  to 
provide  an  early  deployment,  the  need  to 
resolve  an  existing  producibility  risk  prior  to 
Milestone  III,  or  the  need  to  minimize  very 
uneconomical  stand-down  periods  for 
production  facilities  and  personnel.  In  any  of 
these  cases,  however,  the  risk  of  concurrently 
manufacturing  the  additional  units  while  final 
FSD  testing  is  taking  place  must  be  suffic:  ntly 
low  to  warrant  the  increased  financial 
investment. 

DT  of  LRIP  models  is  performed  to  verify 
the  effectiveness  of  improvements  made  as  a 
result  of  T&E  on  the  EDMs,  verify  the 
achievement  of  required  technical  performance 
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specifications,  and  identify  any 
improvements  to  be  incorporated  into  the 
production  version.  The  final  phase  of  DT 
for  the  LRIP  models  is  technical  evaluation 
conducted  in  a  representative  operational 
environment  to  evaluate  readiness  for  OT. 
OT  is  conducted  on  LRIP  models  to  assess 
the  operational  effectiveness  and 
operational  suitability  of  the  system.  The 
final  phase  of  OT  during  FSD  is 
operational  test  and  evaluation  conducted 
in  a  realistic  operational  environment.  OT 
is  usually  conducted  on  the  same  hardware 
as  DT. 

133.4  Production  and  Deployment  Phase 
Activity 

The  transition  from  FSD  to  Production 
translates  the  engineering  model  into 
production  hardware  for  delivery  to  the 
user.  The  objectives  of  the  production 
effort  are  to  achieve:  1)  production  of 
authorized  quantities  on  schedule  and 
within  budget;  and  2)  readiness  for  system 
deployment.  The  objectives  of  the 
deployment  effort  are  to  achieve  a  high 
level  of  operational  readiness  for  the 
deployed  system,  that  is,  personnel 
assignment,  training,  maintenance, 
supply/spare  support  and  overhaul, 
alteration,  and  repair.  The  Production  and 
Deployment  Decision  (Milestone  III)  is  the 
decision  to  produce  systems  for  permanent 
installation  on  fielded  units,  land-based 
configuration,  and  training  facilities,  >r  for 
inventory.  The  production  phase  entails 
(as  required)  both  development  and 
operational  testing.  Activity  is  focused  on 
production  qualification  testing  of 
production  items.  Additional  limited 
development  test  and  evaluation  is 


conducted  after  Milestone  III,  usually  to: 

a.  Verify  the  effectiveness  of  product 
improvements  or  corrections  made  after  the 
FSD  technical  and  operational  testing. 

b.  Demonstrate  the  adequacy  of  redesigns 
as  a  result  of  production  problems  or  early 
follow-on  operational  test  and  evaluation. 

Effective  data  collection  and  analysis  are 
essential  components  of  a  successful  T&E 
program.  Assessment  and  evaluation  of  a 
system  are  continuing  efforts  throughout 
production  and  deployment.  Future  T&E 
efforts  depend  heavily  on  lessons  learned  from 
previous  procurements,  and  ways  to  improve 
performance  must  continually  be  sought. 
Thus,  an  effective  T&E  program  must 
incorporate  continuing  evaluation  of  a  newly 
procured  system  while  it  is  deployed  in  its 
operational  environment. 

Successful  ongoing  assessment  requires  a 
means  to  identify,  report,  and  analyze  the 
cause  of  all  failures  and  provide  for 
appropriate  corrective  action  such  as  design 
changes,  manufacturing  process  changes,  and 
improved  quality  control. 

Reliability  assessment  is  primarily 
concerned  with  problem  identification  and 
evaluations  to  assess  operational  performance. 
Data  should  be  collected  on  all  technical 
problems  and  failures,  which  should  be 
classified  as  critical,  major,  and  minor;  failures 
L'^ould  also  be  evaluated  as  relevant  or 
non-relevant.  Relevant  failures  are  defined  as 
failures  in  the  system  being  tested;  other 
failures  are  considered  non-relevant  (e.  g.,  if 
the  test  equipment  breaks  during  testing). 
Relevant  failures  require  retesting,  whereas 
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testing  can  continue  after  repair  of  a 
non-relevant  failure.  Finally,  careful 
management  of  data  generated  through 
corrective  activities  is  essential. 

13.4  MANAGEMENT  OF  TEST 
PROGRAMS 

The  PM,  through  the  systems  engineer, 
must  coordinate  testing  conducted  by 
contractors,  subcontractors,  laboratories, 
and  field  activities,  and  must  maintain  a 
real-time  network  that  provides  all  the 
proper  information  with  which  to  make 
engineering  and  program  decisions.  The 
TEMP  can  be  an  effective  tool  to 


document  this  network,  to  publicize  the 
reporting  procedures,  and  to  identify 
contingency  plans  and  resources. 

Regardless  of  the  size  of  the  program 
office  staff,  someone  should  be  directly 
responsible  for  focusing  appropriate 
management  attention  on  the  T&E  program, 
and  that  individual’s  name  should  be 
publicized  as  a  point  of  contact.  Experience 
has  shown  that  the  first  critical  task,  and  the 
one  that  should  remain  prominent  as  the 
program  progresses,  is  to  identify  and 
assemble  the  necessary  T&E  engineering 
support  at  each  of  the  locations  of  major 
program  activity.  The  requirements  will  vary 
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as  the  program  passes  through  the 
different  phases.  Keeping  a  good  balance 
of  such  support  and  maintaining  well- 
defined  working  relationships  requires 
significant  program  office  attention, 
frequently  more  than  anticipated. 

13.5  TEST  PROGRAM  REPORTING 

A  graphic  summary  of  test  reporting 
paths  is  shown  in  Figure  13-5.  As 
described  in  DoDD  5000.3  [1],  these 
formal  T&E  reporting  requirements 
prepared  by  major  T&E  agencies  include 
the  following  for  the  Director,  OT&E. 

a.  DOT&E  ASSESSMENT  -  Prepared 
for  each  formal  review  of  a  major  system 
under  development,  this  assessment  is  a 
report  of  the  operational  effectiveness  and 
suitability  of  the  system,  based  on  the 
results  of  operational  testing. 

b.  LRIP  Report  -  Prepared  for  the 
Secretary  of  Defense  and  Congress,  this 
report  addresses  whether  OT&E  on  initial 
production  items  was  adequate,  and 
whether  results  confirm  the  effectiveness 
and  suitability  of  the  items. 

c.  Annual  OT&E  Report  -  Submitted  to 
Congress,  this  report  summarizes  all  OT 
activities  within  DoD  for  the  fiscal  year, 
including  recommend'itions  on  the  status 
and  adequacy  of  OT  resources  and  funding 
levels. 

For  the  DDDRE(T&E;  mal  T&E 
reporting  requirements  include: 

a.  Defense  Acquisition  Board  (DAB) 


Assessment  -  Provided  at  each  formal  system 
review,  this  report  contains  a  detailed 
technical  assessment  of  the  system’s 
performance. 

b.  Annual  Foreign  Weapons  Evaluation 
Report  -  This  is  an  annual  report  to  Congress 
on  NATO  systems/  equipment  evaluated 
during  the  previous  fiscal  year. 

c.  Joint  Test  and  Evaluation  Program 
Reports. 

DoD  components  are  responsible  for 
providing  documentation  including 
end-of-phase  test  reports  (both  DT  and  OT), 
reports  of  significant  T&E  events  (such  as 
missile  launches  or  live  firings),  annual  OT 
reports,  T&E  financial  reports,  and 
information  for  congressional  data  sheets  and 
contract  award  reports. 

The  PM  must  support  the  needs  of  these 
and  other  formal  reporting  requirements.  In 
addition,  detailed  test  reports  must  be 
available  to  substantiate  the  conclusions  and 
recommendations  of  the  PM. 

At  every  level  of  the  organization,  there 
are  many  incentives  to  minimize  formal 
written  reporting.  It  is  expensive  and 
time-consuming,  and  its  full  value  is  rarely 
understood  by  the  developer  of  the  report. 
But  written  reporting  is  an  absolute  necessity 
as  a  vital  communication  medium  among  the 
organizations  involved  in  the  program  and 
among  the  echelons  within  each  organization. 
Test  reports  are  also  important  historical 
documents.  Frequently,  reports  from  earlier 
test  events  are  recalled  during  development  of 
engineering  changes  to  the  system. 
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13.6  RELATION  OF  T&E  TO  SYSTEMS 
ENGINEERING  DOCUMENTATION 

The  role  of  the  systems  engineering 
process  and  its  related  documentation  is 
fully  described  in  Chapters  6,  7,  and  8  of 
this  guide.  The  identification  of  test 
requirements  is  an  integral  part  of  the 
systems  engineering  process.  Test 
requirements  primarily  appear  on  RASs, 
Concept  Description  Sheets  (CDSs),  and 
TRSs.  Testing  requirements  also  surface 
in  the  WBS,  statement  of  work,  the  system 
specification,  and  the  Integrated  Logistics 
Support  Plan  (ILSP)/  Logistics  Support 
Analysis  (LSA)  documents.  The  TRS  is 
the  principal  test  requirements  document 
and  provides  a  primary  input  to  the  TEMP 
as  well  as  the  test  requirements  section  of 
specifications  of  all  types.  The  TRS  is 


described  in  detail  in  paragraph  13.9.  The 
flow  of  test  requirements  in  the  principal 
systems  engineering  documentation  is  shown  in 
Figure  13-6. 

13.7  RELATION  OF  T&E  TO  THE 
TECHNICAL  PERFORMANCE 
MEASUREMENT  PROGRAM 

The  technical  performance  measurement 
(TPM)  program,  described  in  Chapter  14  of 
this  guide,  must  be  considered  as  an  integral 
part  of  the  T&E  program.  TPM  is  defined  as 
produce  design  assessment  and  forms  the 
backbone  of  the  development  testing  program. 
It  estimates,  through  engineering  analyses  and 
tests,  the  values  of  essential  performance 
parameters  of  the  current  design  and  serves  as 
a  major  input  in  the  overall  evaluation  of 
operational  effectiveness  and  suitability. 
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13.8  RELATION  OF  T&E  TO  THE 
SOFTWARE  DEVELOPMENT 
PROCESS 

Chapter  20  of  this  guide  describes  the 
specialized  role  of  test  and  evaluation  in 
the  software  development  process. 
Became  of  unique  demands  imposed  by 
software,  special  procedures, 
documentation,  and  design  reviews  are  set 
out  for  test  and  evaluation  of  software. 
For  clarity,  this  material  has  not  been 
repeated  in  this  chapter.  However,  it 
should  be  considered  an  integral,  vital  part 
of  the  overall  T&E  program. 

13.9  DOCUMENTATION 

13.9.1  Test  and  Evaluation  Master  Plan 

The  Test  and  Evaluation  Master  Plan 
(TEMP)  is  in  a  five  part  format  as  detailed 
in  DoD  5000.3-M-l  [2].  Briefly,  Part  I 
concerns  system  details  including 
production  delivery  information  and  the 
operational  and  technical  goals  and 
thresholds.  Part  II,  program  summary, 
includes  a  schedule  chart  that  provides  an 
overview  of  the  major  acquisition  and  T&E 
events.  Parts  III  (DT  Outline)  and  IV 
(OT  Outline)  describe  in  quantitative  terms 
the  scope  of  each  major  test  period.  Part 
V,  the  Test  Resource  Summary,  identifies 
special  resources  required  for  the  test 
program  and  when  those  resources  will  be 
needed.  The  format  must  be  adhered  to 
as  closely  as  possible,  to  assist  the  many 
organizations  that  utilize  the  TEMP  as  a 
source  of  selected  information  for  their 
planning,  resource  allocation,  and  facilities 
management.  The  specified  format  is 
shown  in  Figure  13-7. 


The  TEMP  is  a  dynamic  document  with 
contents  that  should  be  factual  and  specific, 
avoiding  generalities  and  emphasizing 
quantifiable  and  testable  requirements,  both 
operational  and  technical.  Although  a 
summary  document,  it  is  imperative  that 
pertinent,  but  integrated,  facts  and  descriptions 
be  included.  The  contents  must  describe  the 
amount  and  type  of  testing  to  be  conducted 
before  each  milestone,  and  the  resources 
required.  The  PM  is  responsible  for 
developing  the  TEMP,  including  its  content 
and  preparation.  However,  since  Part  IV 
concerns  operational  T&E,  DoD  components 
usually  require  the  Operational  Test  Agency 
to  be  responsible  for  the  preparation,  content, 
and  coordination  of  that  part  of  the  TEMP. 
Therefore,  the  PM  must  establish  early  liaison 
with  the  OTA  to  ensure  an  integrated 
approach  to  the  TEMP. 

Part  I  -  System  Details 

Part  I  contains  the  mission  and  system 
descriptions;  the  system  objective  in  terms  of 
mission  or  operational  requirements;  and  the 
statement  of  operational  and  technical  goals 
and  thresholds. 

The  system  description  is  a  brief, 
mission-oriented  statement  describing  the  new 
or  improved  capability  that  this  system  is 
intended  to  provide.  It  must  be  traceable  to 
systems  engineering  documentation  such  as  the 
FFBD  and  RAS.  While  it  must  be  concise, 
the  system  description  is  written  with  the 
assumption  that  most  TEMP  readers  lack 
extensive  familiarity  with  the  program.  Key 
functions  and  interfaces  with  other  systems 
must  be  described.  Again,  this  material  must 
be  correlated  with  the  systems  engineering 
documentation  and  the  system  level 
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Typical  TEMP  Format 
Figure  13-7 


Part  I-  System  Description 

1.  Mission  Description 

2.  System  Description 

3.  Required  Technical  Characteristics 

4.  Required  Operational  Characteristics 

Part  II-  Program  Summary 

1 .  Management 

2.  Integrated  Schedule 

Part  III-  DT&E  Outline 

1.  Critical  Technical  Characteristics 

2.  DT&E  to  Date 

3 .  Special  Requirements  for  System/  Subsystem  Retest 

4 .  Future  DT&E 

Part  IV-  OT&E  Outline 

1.  critical  Operational  Issues 

2 .  OT&E  to  Date 

3 .  Future  OT&E 

Part  V-  Test  and  Evaluation  Resource  Summary 

1.  Test  Articles 

2.  Test  Sites  and  Instrumentation 

3 .  Test  Support  Equipment 

4 .  Threat  Systems 

5 .  Test  Targets 

6.  Operational  Force  Test  Support 

7.  Simulators,  Models,  and  Testbeds 

8.  Special  Requirements 

9.  T&E  Funding  Requirements 

10.  Resource  Schedule 

11.  Manpowver/  Training 

Appendices 
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specification  (Type  A  specification).  If  the 
system  appears  to  be  similar  to  systems 
currently  in  service,  this  paragraph  should 
indicate  the  ways  that  it  differs.  Part  I 
must  relate  directly  to  the  mission  need 
statement  (MNS),  acquisition  strategy, 
systems  engineering  data  package,  and  the 
ILSP.  Each  of  these  documents  contains 
data  that  augment  the  planned  system 
operational  concept. 

Part  I  contains  an  abbreviated, 
functional  system  description  including  the 
key  functions  of  the  system,  the  inter  A  and 
intra-system  interfaces,  as  well  as  unique 
sy^-m  characteristics.  When  practical,  key 
functions  should  include  a  mission/  function 
matrix  showing  the  relation  between 
primary  functional  capabilities  that  must  be 
demonstrated  through  testing  of  the 
operational  mission(s)  and  concept(s). 
Part  I  must  include  statements  concerning 
unique  characteristics  that  could  lead  to 
special  test  requirements  such  as  nuclear 
hardness,  data-link  vulnerability,  and 
electromagnetic  emission  control.  The 
remaining  Part  I  elements  are  performance 
oriented  characteristics  and  issues.  Each 
paragraph  should  contain  information  on 
characteristics  that  must  be  achieved  to 
avoid  jeopardizing  program  viability  or 
success.  In  most  cases,  these  items  should 
include  reliability,  availability,  and 
maintainability  as  identified  in  the  decision 
milestone  documentation. 

The  required  characteristics  are  those 
areas  of  technical  and  operational  risk  that 
must  be  resohed  by  test  and  evaluation 
and  that  are  critical  to  the  success  of  the 
program.  For  major  acquisition  programs, 
they  can  be  derived  from  the  System 


Concept  Paper  (SCP),  Decision  Coordinating 
Paper  (DCP)  or  other  decision  process 
documentation.  In  addition  to  identifying 
required  characteristics,  the  specific  test  events 
in  Parts  III  and  IV  that  address  each  issue 
should  be  cross-referenced.  Issues  resolved 
through  test  and  evaluation  are  dropped  from 
Part  I  and  discussed  under  results  of  tests  to 
date  in  Part  III  and  IV,  as  appropriate.  If 
and  when  new  issues  surface,  they  should  be 
added  to  Part  I. 

Operational  and  technical  thresholds 
include  specific  performance  requirements. 
As  stated  in  this  section,  these  are  the 
requirements  against  which  the  system 
ultimately  will  be  evaluated.  It  is  important 
that  these  thresholds  be  traceable  through  the 
systems  engineering  documentation  and 
correlated  to  the  TPM  program.  Thresholds 
are  the  minimum  acceptable  value,  consistent 
with  the  operational  requirements,  that  must 
be  demonstrated  for  program  approval  and 
continuation.  To  the  extent  that  these 
requirements  are  loosely  defined,  the 
operational  evaluation  will  also,  of  necessity, 
be  subjective.  If  test  planning  is  poorly  done, 
the  PM  and  the  operational  test  director  could 
have  different  views  about  the  expected  system 
performance  and  capability.  The  PM  and  the 
OTA  must  develop  quantifiable,  testable 
thresholds  and  include  them  in  the  TEMP. 
The  thresholds  that  are  of  major  concern  (the 
most  demanding)  must  be  included. 

In  some  cases,  there  can  be  overlap 
between  the  development  test  and  operational 
test  thresholds.  In  general,  an  operational  test 
threshold  reflects  what  the  unit  with  the 
equipment  or  system  will  do,  while  the 
development  test  threshold  reflects  how  well 
the  equipment  or  system  will  perform  the  task. 
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Failure  criteria  for  the  reliability  thresholds 
must  be  delineated  and  agreed  upon  by 
both  the  PM  and  operational  test  director. 
The  failure  criteria  identify  what  is  counted 
as  a  failure,  as  well  as  the  classification  of 
failures  as  critical  mission  preventing, 
major  mission  degrading,  or  minor  failures. 
If  the  configuration  of  the  system  involves 
software,  these  criteria  can  be  equated  to 
software  failure. 

Part  II  -  Program  Summary 

Part  II,  management  is  addressed  to 
identifying  responsible  T&E  organizations 
in  conjunction  with  the  T&E  strategy 
needed  to  support  the  overall  program 
acquisition  strategy.  Define  quantities  of 
production  articles  (LRIP)  necessary  to 
support  T&E.  Discuss  constraints  on 
schedule,  budget  or  other  resources  which 
may  impact  T&E  activities. 

Part  II,  the  integrated  schedule  chart, 
can  be  a  one-page  chart  indicating  the 
major  development  test  and  operational 
test  periods  and  their  relationship  to  the 
decision  milestone,  test  article  availability, 
and  production  deliveries.  Most  errors 
that  appear  in  this  chart  are  the  result  of 
either  missing  information  or  faulty 
sequencing  (such  as  a  test  beginning  before 
test  articles  are  available,  test  reports  due 
before  completion  of  tests,  or  a  milestone 
decision  shown  prior  to  the  schedule 
completion  of  the  test  that  provided  the 
results  to  support  the  decision).  Undue 
optimism  frequently  leads  to  a  milestone 
decision  shown  concurrent  with  the 
completion  of  testing.  The  schedule 
should  reflect  a  realistic  sequence  of  events 
with  adequate  time  to  allow  for  document 


preparation  and  the  necessary  review  cycles. 
Part  III  -  DT&E  Outline 

Part  III,  the  DT  outline,  includes  a 
description  of  each  major  DT  period,  stating 
briefly  the  configuration  of  the  system  being 
tested,  the  objective  of  the  test  period,  and 
the  scope  of  the  testing.  For  the  sake  of 
brevity,  a  description  of  past  T&E  periods 
may  be  combined  and  need  not  emphasize 
quantifications  in  the  scope  of  testing. 
However,  the  objectives  achieved  and 
summary  conclusions  drawn  should  be 
included.  Part  III  also  discusses  scheduled 
pre-production  qualification  test  and 
evaluation,  production  qualification  test  and 
evaluation,  and  any  requirements  for  system  or 
subsystem  retest. 

Part  IV  -  OT&E  Outline 

Part  IV,  the  operational  test  outline,  is 
usually  provided  by  the  OTA  evaluator.  In 
some  cases,  particularly  during  the  D/V  phase 
and  early  in  the  FSD  phase,  demonstration  of 
selected  test  requirements  might  be  met  by 
concurrent  development  and  operational 
testing,  or  combined  development  and 
operational  testing.  Concurrent  or  combined 
testing  is  considered  where  the  use  of 
development  test  results  in  operational  testing 
reduces  total  test  time  and  cost.  Such  a 
strategy  must  be  agreed  to  in  advance  by  both 
the  Program  Manager  and  the  operational 
evaluator  and  must  be  documented  in  this  part 
of  the  TEMP  as  well  as  in  the  acquisition 
strategy.  In  any  case,  close  coordination  with 
the  program  managers  test  director  and  the 
OTA  will  be  necessary  to  ensure  adequate 
integration  of  the  operational  test  phase  with 
the  remainder  of  the  test  program,  and  with 
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the  rest  of  the  TEMP. 

Part  V  -  Test  and  Evaluation  Resource 
Summary 

Part  V,  the  T&E  resource  summary,  is 
often  neglected.  Essentially,  this  part 
identifies  special  resources  and  facilities 
needed  to  execute  the  T&E  program.  The 
entries  in  this  summary  should  reflect, 
where  applicable,  resources  such  as  type  of 
aircraft,  number  of  flight  hours  required, 
class  of  ship,  battlefield  threat  simulators, 
and  threat  aircraft  required  for  operational 
realism.  Part  V  identifies  the  quantities, 
types,  and  configurations  of  test  articles, 
the  major  ground-based  test  sites,  climatic 
laboratories,  special  gunnery  ranges,  or 
unusual/special  real  estate  considerations. 
This  part  of  the  TEMP  should  include  any 
special  equipment  installation  and  removal 
schedules,  and  personnel  training  and 
travel  requirements.  The  systems 
engineering  process  should  be  used  to 
define  these  requirements. 

The  two  most  common  errors  in 
preparing  the  Test  Resource  Summary  are 
omissions  and  inconsistencies.  Omissions 
commonly  include  ammunition 
requirements  that  are  above  and  beyond 
the  requirements  for  training.  Ammunition 
typically  can  require  30  or  more  months  to 
obtain.  Inconsistencies  include  test  period 
differences  due  to  poor  and  inadequate 
schedule  control.  Planning  for  the  use  of 
ranges,  test  facilities,  and  targets  should  be 
included  in  this  part.  The  capabilities  of 
frequently  used  facilities  should  be 
consulted.  Readily  available  documents 
include  information  concerning  the 
commonly  used  aerial,  surface,  and 


subsurface  targets,  as  well  as  data  collection, 
storage,  and  reduction  requirements. 

13.9.2  Test  Requirements  Sheet 

The  Test  Requirement  Sheet  (TRS),  shown 
in  Figure  13-8,  an  Army-peculiar  document,  is 
an  example  of  a  worksheet  that  serves  several 
purposes  in  the  systems  engineering  process. 
It  identifies  all  the  requirements  that  TTluSi  be 
demonstrated  or  verified  during  the  life  cycle 
testing.  The  TRS  serves  as  a  tool  for 
management  to  check  whether  appropriate 
provisions  have  been  made  for  verification  of 
all  performance/  design  requirements.  It  also 
provides  for  the  identification  of  test  functions 
for  the  test  cycle  of  the  systems  engineering 
process.  The  TRS  is  used  to  describe  test 
requirements  of  the  overall  system.  By 
appropriate  repetition,  the  TRS  is  indentured 
to  the  level  desired  (e.g.,  end  item,  assembly, 
sub-assembly,  or  component). 

The  TRS  for  a  system,  item,  or  assembly 
contains  information  referencing  each  test 
requirement  from  the  system,  development,  or 
product  (Types  A,  B,  and  C  respectively) 
specifications  that  is  subject  to  verification.  It 
also  identifies  the  verification  method  and  the 
type  of  testing  fas  discussed  in  paragraph 
13.3). 

For  each  system,  end  item,  assembly, 
subassembly,  or  component  for  which  the 
verification  method  is  designated  in  column  2, 
functional  analysis  (using  RAS,  FFBDs,  and 
TLS),  synthesis  (using  Schematic  Block 
Diagrams  (SBDs)  and  Concept  Description 
Sheets  (CDSs),  evaluation  and  decision  (using 
Trade  Study  Reports  (TSR)),  and  description 
(using  Design  Sheets  (DS))  are  required  in 
order  to  ensure  the  timely  availability  of  the 
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TEST  Cl  NO. 

REQUIREMENT  NOMENCLATURE  DETAIL  SPEC.  NO.  (OR  OTHER  IDENTIFICATION) 

SHEET 


Figure  13-8 

Test  Requirements  Sheet 


test  elements.  Test  elements  stated  in  the 
specifications  are  the  source  of 
performance  requirements  for  test 
functions.  Test  equipment,  facilities, 
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CHAPTER  14 


TECHNICAL  PERFORMANCE  MEASUREMENT 


14.1  INTRODUCTION 

The  Program  Manager  (PM)  must  be 
cognizant  of  three  basic  program  elements: 
1)  cost,  2)  schedule,  and  3)  technical 
performance.  The  first  two  are  tracked 
through  cost  and  schedule  control  systems. 
The  last  item  is  tracked  through  the  technical 
performance  measurement  (TPM)  system 
defined  in  MIL-STD-499A  [1]. 

TPM  is  defined  as  the  product  design 
assessment  which  estimates,  through 
engineering  analyses  and  tests,  the  values  of 
essential  performance  parameters  of  the 
current  design  of  work  breakdown  structure 
(WBS)  product  elements.  TPM  is  used  to  1) 
forecast  the  values  to  be  achieved  through 
the  planned  technical  program  effort,  2) 
measure  differences  between  the  achieved 
values  and  those  allocated  to  the  product 
element  by  the  systems  engineering  process, 
and  3)  determine  the  impact  of  these 
differences  on  system  effectiveness.  The 
purpose  of  TPM  is  to: 

a.  Provide  visibility  of  actual  versus 
planned  performance 

b.  Provide  early  detection  or  prediction  of 
problems  which  require  management 
attention 

c.  Support  assessment  of  the  program 
impact  of  proposed  change  alternatives. 


Use  of  TPM  alerts  program  management  to 
potential  performance  deficiencies  before 
irrevocable  cost  or  schedule  impact  occurs. 
Where  a  program  also  has  an  overall  risk 
assessment  program,  TPM  provides  data  for 
technical  risk  planning  and  assessment.  Input 
from  the  risk  management  process  will  also 
assist  in  determining  parameter  criticality  in 
the  TPM  selection  process. 

TPM  takes  selected  critical  product 
elements  of  the  contract  work  breakdown 
structure  (CWBS)  and  provides  a  method  of 
monitoring  their  technical  status  throughout 
the  program.  In  implementing  TPM,  values 
of  measurement,  illustrated  in  Figure  14-1, 
are  defined  as  follows: 

a.  Planned  Value  -  The  anticipated  value 
of  a  parameter  at  a  given  point  in  the 
development  cycle.  A  plot  of  planned  value 
versus  time  is  known  as  the  planned  value 
profile.  It  may  be  desirable  to  indicate  a 
range  of  acceptable  values  versus  time. 
When  this  range  is  shown,  it  is  known  as  a 
tolerance  band. 

b.  Demonstrated  Value  -  The  value 
estimated  or  measured  in  a  particular  test  or 
analysis. 

c.  Specification  Requirement  -  The  value 
or  range  of  values  contained  in  a  contractual 
development  specifications  (Type  B)  or 
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Figure  14-1 
TPM  De f i n i t ions 
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allocated  from  such  a  specification,  with  a 
verification  requirement  for  the  end  product. 

d.  Current  Estimate  -  The  value  of  a 
parameter  predicted  for  the  end  product  of 
the  contract. 

e.  Demonstrated  Technical  Variance  -  The 
difference  between  the  planned  value  and 
the  demonstrated  value  of  a  parameter. 

f.  Predicted  Technical  Variance  -  The 
difference  between  the  specification 
requirement  and  the  current  estimate  of  the 
parameter. 

Technical  parameters  to  be  reported  and 


tracked  are  determined  through  identification 
of  technically  critical  areas  from  review  of 
systems  engineering  documentation, 
development  specification  (Type  B) 
requirements,  and  planned  contractual 
performance  incentives  and  their  relationship 
to  system  measures  of  effectiveness. 
Parameters  to  be  tracked  by  the  contractor 
should  be  identified  in  the  contract.  At  the 
completion  of  each  evaluation,  results  are 
recorded  for  comparison  with  planned  values. 
Variances  in  results  from  planned  values  are 
analyzed.  The  analysis  includes  evaluation  of 
the  effect  of  variances  on  the  technical 
program  risk,  schedule,  and  cost.  Summary 
performance  status  reports  are  prepared 
from  the  basic  parameter  status  data 
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provided  by  TPM.  The  information  flow  for 
assessment  is  illustrated  in  Figure  14-2.  It 
includes  TPM  work  breakdown  elements, 
master  parameter  list,  planned  parameter 
profiles,  summation  models,  parameter  status 
tracking  and  forecast,  records  of  achieved 
parameter  profiles,  summary  performance 
status  report,  and  problem  analysis  and 
corrective  action.  The  first  four  items  are 
the  outputs  of  planning  and  replanning 
efforts.  They  form  the  inputs  to  technical 
performance  measurements  and  assessments. 
These  four  items  can  be  initially 
accomplished  during  the  Concept 
Demonstration/  Validation  (D/V)  phase  of 
the  program. 


14.2  PLANNING  FOR  TPM 

During  the  preparation  of  the  D/V  phase 
requests  for  proposal  (RFPs),  the  program 
office  must  ensure  that  a  requirement  for  a 
TPM  program  is  specified.  Normally  a  TPM 
program  is  initiated  through  a  requirement 
for  an  Assessment  Plan  ?:■  part  of  the  System 
Engineering  Management  Plan  (SEMP).  The 
Assessment  Plan  identifies  the  parameter 
selection  process  and  parameters  that  have 
been  selected  at  the  time  of  plan  submittal. 
To  the  extent  parameters  have  been 
identified,  the  plan  forecasts  the  values  to  be 
attained  through  the  development  program 
and  the  methods  to  be  used  for  assessing  and 


Figure  14-2 

TPM  and  System  Effectiveness 
Assessment  Information  Flow 
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forecasting.  For  each  scheduled  assessment, 
the  plan  specifies  the  conditions  associated 
with  conducting  tests  or  other  evaluations, 
and  the  measurement  methodology.  In 
summary,  a  typical  Assessment  Plan  contains 
the  following  information: 

a.  A  summary  of  the  plan  for 
demonstration  of  system  technical 
performance 

b.  Lists  of  all  critical  configuration  items 
(CIs),  their  work  breakdown  structure  (WBS) 
numbers,  and  specification  numbers;  the  key 
parameters  for  each  Cl;  and  numbers  of 
specification  paragraphs  which  completely 
identify  these  key  parameters  and  their 
quantitative  requirements 

c.  Lists  of  milestones  related  to 
performance  achievement  for  the  system  and 
each  subsystem  by  titles  and  schedule 
milestone  numbers 

d.  Lists  of  the  numbers  of  Test 
Information  Sheets  for  those  performance 
achievement  events  (PAEs)  that  are  tests. 

TPM  can  begin  when  configuration  item 
requirements  allocation  is  substantially 
complete  (when  draft  Type  B  specifications 
are  available).  Normally  this  would  happen 
during  the  D/V  phase.  A  good  key  for 
timing  the  initiation  of  a  TPM  effort  would 
also  be  when  the  majority  of  configuration 
item  Preliminary  Design  Reviews  (PDRs) 
have  been  completed.  It  is  only  at  this  point 
that  a  reliable  and  stable  allocated 
performance  base  for  WBS  product  elements 
is  available.  A  TPM  program  continues  until 
testing  has  proven  that  design  is  complete. 
TPM  is  extended  into  the  Production  phase 


whenever  there  are  major  modification  or 
product  improvement  programs  involving  Cl 
design  changes. 

If  product  design  is  not  stable  during 
D/V,  TPM  planning  may  start,  but  initiation 
of  assessment  and  reporting  could  be 
postponed  until  the  early  part  of  Full  Scale 
Development  (FSD).  Another  option  is  to 
begin  only  on  stable  parts  of  the  WBS  and 
withhold  implementation  on  selected 
elements  that  may  be  more  unstable. 
However,  in  most  cases  this  is  not  desirable 
since  it  does  not  provide  valid  system  level 
reporting. 

143  SELECTING  TPM  PARAMETERS 

The  selection  of  parameters  for  TPM 
begins  with  a  review  of  contract  performance 
incentives,  performance  specifications,  and/or 
systems  engineering  documentation  that 
details  critical  performan  i  elements. 
Ordinarily,  a  review  of  the  relationship  of 
these  elements  to  system  measures  of 
effectiveness  will  be  necessary  to  identify  or 
confirm  parameter  criticality.  Parameters 
selected  for  tracking  should  be  key  indicators 
of  program  success.  In  developing  a 
cost-effective  TPM  program,  only  a  few  key 
parameters  should  be  selected  at  the  top 
level  because  the  total  number  of  detailed 
parameters  tracked  to  support  these  key 
parameters  may  be  10  times  greater. 
Parameters  can  be  related  to  hardware, 
software,  logistics,  ’’uman  factors,  that  is,  to 
any  product  or  functional  area  of  this  system. 
Typical  TPM  parameters  include  those  listed 
in  Figure  14-3.  Because  an  increasing 
amount  of  system  development  resources  are 
related  to  software  development,  and  because 
software  often  poses  the  greatest  technical 
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Figure  14-3 

Typical  TPM  Parameters 
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risk,  PMs  should  be  particularly  sensitive  to 
the  selection  of  parameters  from  this  area. 
The  parameters  selected  should  be  limited  by 
the  following  criteria: 

a.  They  are  the  most  significant  qualifiers 
or  determinants  of  the  total  system  product. 

b.  A  direct  measure  of  value  can  be 
rapidly  derived  from  resuits  of  analyses  or 
tests. 

c.  Time-phased  values  and  tolerance  bands 
can  be  predicted  for  each  parameter  and 
substantiated  during  design,  development,  and 
test. 

The  framework  for  a  TPM  parameter 
tree  is  developed  from  the  CWBS,  described 
in  Chapter  9  of  this  guide.  The  CWBS 
defines  all  tasks  to  be  accomplished  on  the 
program  and  follows  the  system  hierarchy 
and  program  specification  tree.  Principally, 
the  product  elements  of  the  CWBS  are 
applicable  to  TPM.  Some  items,  such  as 
support  equipment,  may  not  contribute 
significantly  to  the  selected  TPM  parameters. 
However,  some  elements  may  contribute  to 
several  parameters,  for  example,  weight, 
power,  reliability,  and  error  from  a  guidance 
component. 

The  TPM  parameter  tree  defines  the 
build-up  of  system  parameter  summaries 
from  selected  elements  of  the  CWBS. 
Where  selected  parameters  share  a  common 
WBS  element,  a  single  tree  can  define  the 
TPM  schema.  Wheie  differences  cannot  be 
easily  shown  by  coding,  more  than  one  tree 
may  be  required  for  the  overall  program. 

The  linkage  between  the  CWBS  and  the 


TPM  parameter  tree  is  the  key  to  the 
parameter  tree’s  use  as  an  effective 
management  tool.  Developing  the  TPM 
parameter  tree  from  the  CWBS  ensures 
traceability  of  progress  on  technical 
performance  to  cost  and  schedule  aspects  of 
the  work  effort  (through  the  cost/  schedule 
control  system).  Program  management  can 
then  associate  technical  performance 
variances  (such  as  a  weight  parameter 
exceeding  the  tolerance  limit  by  10  percent) 
with  schedule  and  budgetary  status  (80 
percent  of  budget  expended,  less  than  one 
month  for  final  value  to  be  achieved). 

Note  that  the  TPM  parameter  tree  may 
not  perfectly  correspond  to  the  CWBS  in 
content  or  degree  of  detail.  For  example,  in 
solving  a  compl'  *  -oftware  problem,  it  may 
be  necessary  to  expand  certain  parts  of  the 
WBS  to  facilitate  parameter  tracking  in  that 
area. 

Figure  14-4  illustrates  a  partial  tree  for  a 
shipboard  combat  system  electronic  suite. 
It  illustrates  how  requirements  can  be  tracked 
at  several  levels.  In  the  example,  availability 
requirements  are  set  and  tracked  at  the  ship 
and  combat  system  level.  However,  these 
summaries  are  products  of  mean  time 
between  failures  and  mean  time  to  repair 
(MTBF  and  MTTR)  requirements  set  by  the 
specification  for  the  weapons  control 
package.  Requirements  for  MTBF  and 
MTTR  are  set  by  requirements  imposed  on 
configuration  items  such  as  the  weapons 
control  computer  program  or  signal  data 
converter.  During  the  planning  phase,  one 
or  all  of  these  levels  could  be  chosen  for 
tracking,  depending  on  the  criticality,  and 
perhaps  the  inherent  technical  risk,  of  the 
parameter.  The  other  parameter  listed. 
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Figure  14-4 
TPM  Parameter  Tree 


combat  system  reaction  time,  is  perhaps  the 
product  of  weapons  contra’  reaction  time, 
personnel  reaction  time,  etc.  It  may  not  be 
necessary  to  establish  TPM  targets  below 
this  level.  In  all  cases,  the  TPM  parameter 
tree  or  trees  must  be  correlated  to  WBS 
elements.  Figure  14-5  provides  a  TPM  tree 
for  a  fire  control  system. 

The  lower  level  parameters  are  identified 
through  the  requirements  allocation  process. 
These  parameters  represent  allocation  of 
system  level  requirements  to  lower  levels 
within  the  system  hierarchy  and  should  be 
available  in  the  documentation  of  the 
functional  analysis  process  (refer  to  Chapter 
6  of  this  guide). 

The  identification  of  parameters  and  the 
parameter  Assessment  Plan  is  closely  related 


to  the  system  synthesis  described  in  Chapter 
7  of  this  guide.  This  is  the  point  where  end 
item  requirements  allocation  sheets  (RASs) 
are  developed  and  end  item  test 
requirements  sheets  (TRSs)  are  generated. 
The  data  are  then  compiled  into  the  end 
item  (product)  specifications  (Type  C). 

Key  TPM  parameters  must  be  selected 
using  the  full  scope  of  the  systems 
engineering  process.  A  comprehensive  set  of 
key  parameters  should  be  selected  for  the 
system,  for  each  segment,  for  and  for  each 
critical  Cl,  on  the  basis  of  overall  technical 
importance,  technical  risk  assessment, 
parametric  sensitivity  in  the  engineering 
models,  and  interface  relationships.  Top 
summary  level  parameters  and  their 
quantitative  requirements  will  normally 
become  part  of  the  development 
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Figure  14-5 

Shipboard  Fire  Control  System  TPM  Parameters  (Partial) 
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specifications  (Type  B). 

Each  TPM  program  should  establish  the 
requirement  for  a  controlled  Master 
Parameter  List.  The  Master  Parameter  List 
is  usually  developed  by  competing  contractors 
during  the  D/V  phase  and  contains  all 
technical  parameters  that  have  been  selected 
for  measurement  as  part  of  the  TPM 
program.  Where  the  program  office  has 
certain  specific  technical  parameter  reporting 
needs,  the  D/V  phase  RFP  could  contain  a 
partial  list  of  parameters  to  be  included  in 
the  program.  The  RFP  may  also  identify  the 
WBS  level  where  assessment  should  take 
place.  Figures  14-6  and  14-7  illustrate 
Master  Parameter  Lists  for  an  aircraft  and  its 
engine  subsystem. 


14.4  DEVELOPING  PLANNED  PROFILES 

For  each  selected  performance  parameter, 
a  planned  performance  profile  must  be 
established.  As  one  of  the  initial  steps  of  a 
TPM  program,  each  design  group  will 
prepare  appropriate  profiles  using  historical 
data,  test  planning,  contract  requirements, 
etc.  For  those  parameters  that  will  be 
reported  to  the  program  office,  planned 
profiles  are  reviewed  and  approved  by  the 
organization  establishing  the  need  for  that 
data. 

Planned  profiles  may  reflect  constant 
values.  This  would  probably  be  associated 
with  technically  mature,  low  risk  contract  end 
items.  In  this  case,  the  profile  would  appear 
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Figure  14-6 

TPM  Parameters  -  Aircraft 
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Figure  14-7 

-  Aircraft  Engine  Subsystem 
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as  a  horizontal  line  against  time.  This  type 
of  profile  is  illustrated  in  Figure  14-8(a). 
The  requirement  is  the  same  at  each  major 
milestone  for  this  end  item  -  that  is, 
"demonstrate  that  you  have  not  exceeded  the 
specification  weight 

Establishing  realistic  profiles  is  not  usually 
this  simple.  Most  development  items  are 
not  expected  to  reflect  mature  values  during 
initial  analysis  and  testing.  Figure  14-8(b) 
illustrates  that,  historically,  in  the 
development  of  similar  components, 


necessary  design  changes  have  led  to  growth 
of  approximately  15  percent  in  weight.  The 
profile  in  Figure  14-8(b)  compensates  for  this 
growth  by  making  the  initial  requirement 
more  stringent.  Figure  14-8(c)  illustrates 
perhaps  another  variation  of  this  case  where, 
because  of  a  planned  weight  reduction 
program,  the  initial  unit  is  expected  to  be 
heavier  than  the  production  item. 

Planned  profiles  should  not  be  viewed  as 
static,  particularly  where  systems 
engineering/engineeriug  development  is  still 


Figure  14-8 

Planned  Parameter  Profiles 
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Figure  14-9 

Pre-planning  of  TPM  Profile 


in  process.  Where  trade  studies  indicate  that 
cost  or  time  to  achieve  a  planned 
requirement  is  excessive,  the  requirement 
could  be  relaxed  and  new  profiles 
established.  This  will  usually  involve 
adjustment  of  other  parameters  or  parameter 
sets.  Figure  14-9  illustrates  the  replanning  of 
a  profile  based  on  demonstrated  values. 

The  utility  of  all  TPM  assessment  and 
forecasting  methcds  is  usually  enhanced  by 
establishing  a  tolerance  band  as  part  of  the 
planning  process  for  each  profile.  Figure 
14-10  illustrates  the  tolerance  band  for  a 
TPM  parameter  as  it  would  be  indicated  on 
a  TPM  report.  The  boundaries  of  the 
tolerance  band  reflect  the  known  limitations 
of  the  estimating  method  being  used  and  past 
experience.  They  define  the  region  within 


which  it  is  reasonable  to  expect  that  the 
specification  requirement  will  be  achieved 
within  cost  and  schedule  constraints.  Use  of 
both  upper  and  lower  bounds  on  each 
parameter  permits  management  by  exception 
while  providing  notice  of  both 
underachievement  and  overachievement 
trends.  Note  that  the  example  in  Figure 
14-10  shows  a  current  forecast  outside  of  the 
tolerance  band. 

Tolerance  limits  for  performance  profiles 
are  normally  established  by  the  contractor 
during  the  D/V  phase,  with  review  and 
approval  by  the  government  program  office. 
Since  TPM  is  the  Program  Manager’s  early 
warning  system,  this  review  should  include 
top-level  management  review  of  both  the 
profiles  and  tolerance  bands.  An  alarm 
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Figure  14-10 
TPM  Report  Format 
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system  is  valuable  only  if  activation  levels  are 
realistic.  False  alarm  frequency  will  have  a 
major  long  term  impact  on  the  confidence  of 
contractor  and  government  Program 
Managers  in  using  this  tool.  Confident 
management  by  exception  requires  that 
tolerance  band  limits  be  reviewed  at  least 
yearly. 

14.5  ASSESSMENT  METHODS 

There  is  a  wide  range  of  methods 
available  for  assessment  of  technical 
performance.  During  the  D/V  phase,  analysis 
may  be  the  only  way  of  assessing  what  is 
probably  largely  a  paper  design.  There  is 
still  a  range  of  possibilities  in  terms  of  the 
method/  depth  of  analysis  or  degree  of 
simulation. 


Figure  14-11  illustrates  typical  methods 
used  for  both  tracking  and  forecasting 
performance  of  selected  parameters.  The 
actual  method(s)  chosen  for  a  particular 
program  must  be  related  to  such  factors  as 
the  type  of  TPM  parameters  selected, 
maturity  of  program,  etc.  Each  method  has 
associated  costs,  often  requires  development 
time,  and  in  turn  provides  a  specific  level  of 
confidence.  The  government  and  industry 
TPM  managers  must  carefully  trade  off 
program  needs  for  confidence  in  both 
tracking  and  forecasting  with  the  related 
costs. 

During  system  design  and  development, 
achievement  to  date  is  tracked  at  each 
assessment  milestone,  for  each  selected 
parameter,  and  at  each  specified  level  of  the 
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Figure  14-11 
TPM  Methods 


TEST  METHODS 

D/V 

EARLY  FSD 

LATE  FSD 

PRODUCTION 

•  SIMILARITY 

X 

•  ENGINEERING 
ANALYSIS 

X 

.  MODELING 
—  SCALE 

X 

X 

—  MATH 

X 

X 

IF  VALIDATED 

•  TEST 

—  COMPONENT 

X 

—  Cl 

X 

X 

—  SYSTEM 

X 

FORECAST  METHODS 

•  COMPARISON 

X 

X 

•  MATH  MODELS 

X 

X 

•  SIMULATION  MODELS 

X 

X 

X 

•  EXPERIENCE 

X 

X 

X 

•  DECISION 

X 

X 

A 

A 

SDR 

FORMAL  TPM 

PCA 

WBS.  These  point  estimates,  based  on  either 
analysis  or  testing,  are  then  used  to  forecast 
the  expected  value  that  will  be  achieved  at 
the  end  of  the  FSD  phase,  when  the 
production  decision  is  scheduled,  or  the  initial 
operating  capability  (IOC),  as  the  program 
office  chooses. 

If  any  demonstrated  or  forecast  value  falls 
outside  the  planned  tolerance  band, 
corrective  action  plans  are  prepared  by  the 
contractor  and  reviewed  with  the  government 
program  office.  Each  deviation  is  analyzed 
to  determine  cause  and  to  assess  the  impact 
on  higher  level  parameters,  interface 
requirements,  and  system  cost-effectiveness. 


Alternate  recovery  plans  are  developed 
showing  fully  explored  cost,  schedule,  and 
technical  performance  implications.  Where 
performance  exceeds  requirements, 
opportunities  for  reallocation  of  requirements 
and  resources  are  assessed. 

14.6  REPORT  GENERATION 

Parameter  assessment  for  lower  levels  of 
the  WBS  can  often  be  taken  directly  from 
test  data  or  analysis  that  only  involve  one  or 
two  contract  configuration  items.  System 
level  parameters  cannot  be  measured  directly 
until  very  late  in  the  acquisition  cycle  - 
perhaps  long  after  system  responsibility  has 
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shifted  from  the  contractor  to  the 
government.  For  this  reason,  it  is  important 
that  both  contractor  and  government 
management  conduct  a  thorough  review  of 
all  simulation  and  summation  models  used  to 
generate  system  level  TPM  reports  during 
design  development. 

Through  summary  (system  level)  TPM 
reports,  both  the  procuring  agency  and 
contractor  management  may  quickly  identify 
deviations  from  the  planned  parameter 
profile.  The  simplest  example  is  the  system 
weight  parameter.  Total  weight  is  usually  a 
summary  level  parameter,  and  a  simple 
arithmetic  summation  of  the  actual  weight  of 
every  part  of  the  system  must  not  exceed  the 
total  allowable  weight  for  fhe  system  being 
developed.  Each  WBS  product  element 
would  be  allocated  a  specific  or  maximum 
weight.  During  design,  development,  and  test 
phases,  the  weight  of  each  WBS  product 
element  would  be  estimated,  analyzed, 
measured,  reported,  and  then  accumulated 
through  the  simple  summation  model  to 
arrive  at  a  total  system  weight.  Other 
parameters  require  much  more  complex 
summation  models.  Reliability  and 
maintainability  summation  models  require 
information  on  mean  time  between  failure 
and  mean  time  to  repair  of  subsystems  and 
components,  typical  mission  profiles,  intended 
mission  mix,  and  maintenance  manhours  per 
operation  hour. 

To  the  degree  the  government  PM  has 
confidence  in  the  choice  of  parameters, 
method  of  assessment,  and  fidelity  of  data, 
these  reports  will  provide  a  valuable 
management  tool  for  justifying  or  updating 
predictions  of  operational  behavior, 
operational  availability,  and  life  cycle  cost. 


14.7  REPORT  FREQUENCY  AND  TIMING 

The  effective  tailoring  of  reporting 
requirements  is  one  of  the  most  difficult  tasks 
facing  the  government  PM  and  a  major 
objective  of  the  acquisition  streamlining 
initiative  outlined  in  Chapter  10  of  this  guide. 
Reporting  requirements  must  be  tailored 
considering  such  factors  as: 

a.  Program  technical  risk 

b.  Program  complexity 

c.  Program  office  size  r  :d  organization 

d.  Amount  of  engineering  development 

e.  Number  of  (competing)  contractors 

f.  Size  of  program 

g.  Acquisition  phase  of  contract 

h.  Contractors’  track  record. 

One  strategy  that  could  be  employed 
would  be  to  specify  a  minimum  level  of 
formal  reporting,  but  require  that  the 
contractor  utilize  the  TPM  system  as  a 
technical  management  tool  and  maintain 
records  of  internal  actions. 

Whenever  an  engineering  specialty  plan 
is  changed,  or  deviation  from  planned 
technical  performance  values  is  reported, 
traceable  records  must  be  maintained. 
Likewise,  the  procuring  agency  usually 
reserves  the  right  to  require  the  contractor 
to  discuss  the  records  and  TPM  reports  at 
any  level  of  the  WBS  whether  or  not  they 
are  contract  data  requirements  list  (CDRL) 
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items.  The  capability  to  examine,  on  an 
exception  basis,  TPM  data  on  low  level  WBS 
elements  is  crucial  to  the  identification  of 
effective  corrective  actions.  The  availability 
of  this  back-up  information  should  also 
substantially  reduce  the  frequency  with  which 
other  than  top  summary'  level  reports  are 
required  by  the  program  office. 

If  possible,  report  delivery  requirements 
should  be  correlated  with  the  assessment 
points  defined  in  the  planning  profiles.  In 
the  examples  of  Figure  14-8,  each  profile 
specified  three  assessment  points: 
"completion  of  development  testing," 
"completion  of  prototype  testing,"  and 
"production  decision."  TPM  assessment 
points  are  normally  planned  to  coincide  with 
the  planned  completion  of  significant  design 
and  testing  tasks,  program  reviews,  and 
decision  milestones.  These  points  are  also 
prime  candidates  for  contractually  required 


report  deliveries.  Figure  14-12  shows  a 
more  realistic  TPM  report  with  reference  to 
the  related  program  events.  Effective 

streamlining  requires  reporting  schedules 
that  are  carefully  matched  to  the  expected 
level  of  design  activity  affecting  each 
parameter.  Reports  may  be  prepared 

monthly  or  yearly,  as  appropriate. 

Management  by  exception  plans  may  permit 
reporting  of  only  those  parameters  outside 
established  tolerance  limits.  On-line 

automated  transmission  of  TPM  data  is  now 
often  feasible  to  reduce  administrative  costs. 

Typical  candidates  for  TPM  performance 
achievement  and  reporting  milestones  are: 

a.  Design  reviews 

b.  Critical  item  analyses 

c.  Breadboard  tests 


Figure  14-12 
Typical  TPM  Report 
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d.  Hardware  validations 

e.  Performance  tests 

f.  Environmental  tests 

g.  Qualification  tests 

h.  Reliability  tests 

i.  Computer  software  tests 

j.  Integrated  logistic  support  testing  and 
demonstration 

k.  Operational  readiness  testing. 

14.8  TECHNICAL  PROGRESS  REVIEWS 

Periodic  reviews  between  the  government 
and  contractor(s)  are  conducted  to  determine 
whether  the  planned  technical  program 
should  be  altered  as  uncertainties  are 
disclosed,  eliminated,  or  reduced.  These 
reviews,  usually  held  quarterly,  are  a  planned 
part  of  the  systems  engineering  management 
effort,  not  a  reaction  to  technical  problems 
that  suddenly  appear.  They  are  used  to  seek 
opportunities  to  reduce  or  redirect  program 
effort  to  effect  economies  in  budget  and 
time,  as  well  as  to  increase  or  redirect 
program  effort  to  overcome  weaknesses  that 
may  develop  in  the  planned  program.  TPM 
assessment  update  events  for  the 
corresponding  product  elements  of  the  WBS 
may  be  scheduled  to  coincide  with 
configuration  item  design  reviews,  system 
level  technical  reviews,  or  quarterly  technical 
program  reviews.  This  fresh  documentation 
then  forms  the  basis  for  the  review  agenda 
and  analysis. 


14.9  RELATION  OF  TPM  TO  SYSTEMS 
ENGINEERING 

TPM  is  an  essential  element  in  the 
systems  engineering  process.  Demonstrated 
values  that  signal  potential  risks  of  meeting 
specification  requirements  and  design  to  cost 
goals  provide  important  feedback  information 
to  specialty  areas  such  as  reliability  and 
maintainability.  Since  selected  TPM 
parameters  often  overlap  with  principal  trade 
study  criteria,  predicted  technical  variances 
might  impact  the  choice  and/or  weighting  of 
criteria.  Technical  performance  assessment 
depends  on  the  use  of  engineering  analysis, 
test,  and  evaluation  to  make  periodic 
assessments  of  the  status  of  the  technical 
program  in  achieving  the  performance 
parameters  it  has  established  for  the  product 
system.  These  technical  assessments  resulting 
from  TPM,  when  correlated  to  cost  and 
schedule  reports,  provide  the  complete  status 
of  the  program.  They  serve  to  identify  any 
engineering  or  other  technical  problems 
requiring  management  attention,  and  to 
forecast  the  impact  on  program  cost, 
schedule,  and  ultimate  performance  of  any 
out  of  tolerance  conditions. 

14.10  REPORT  FORMAT 

TPM  reports  delivered  to  the  program 
office  are  actually  a  compilation  of  individual 
TPM  parameter  assessments  and  may  be 
prepared  according  to  data  item  description 
(DID)  DI-S-3619  (2].  Each  report  contains 
a  comparison  of  planned  and  actual  values, 
a  design  and  analysis  status,  a  variance 
analysis,  and  supporting  information. 

The  TPM  report  includes  the 
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demonstrated  value,  planned  value,  and 
demonstrated  variance  for  the  design  at  the 
time  of  the  TPM,  plus  the  current  estimate, 
the  current  specification  requirement,  and 
the  predicted  variance  for  the  end  product. 
Determination  of  the  current  estimate  is 
based  on  the  demonstrated  value  and 
changes  to  the  parameter  value  that  can  be 
attained  within  the  remaining  schedule  and 
cost  baseline.  The  performance  comparison 
can  be  in  tabular  or  graphic  form. 

The  TPM  summary  report  might  also 
include  a  status  of  the  design  configuration, 


discussion  of  design  and  engineering 
investigations  (e.g.,  experiments  and  tests 
performed),  analysis  that  supports  the 
demonstrated  value,  and  a  discussion  of 
technical  effort  supporting  the  predicted 
profile  leading  'o  the  current  estimate. 

The  precise  reporting  format  must  be 
tailored  to  program  needs.  The  use  of 
existing  contractor  report  formats  meeting 
the  PM’s  needs  is  an  economy  encouraged 
by  the  acquisition  streamlining  initiative. 
Figure  14-13  illustrates  one  format  for  a 
TPM  summary  status  report. 
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Figure  14-13 

TPM  Summary  Performance  Status  Report 
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CHAPTER  15 


RISK  ANALYSIS  AND  MANAGEMENT 


15.1  INTRODUCTION 

In  the  Department  of  Defense  (DoD) 
context,  risk  can  be  defined  as  a  potential 
occurrence  that  would  be  detrimental  to 
plans  or  programs.  Risk  is  measured  as  the 
combined  effect  of  the  likelihood  of  the 
occurrence  and  a  measured  or  assessed 
consequence  given  that  occurrence.  The  risk 
as  perceived  by  the  Program  Manager  (PM) 
will  be  different  than  that  perceived  by  the 
Systems  Engineer  or  the  user.  Risk  is  viewed 
from  the  perspective  of  the  evaluator,  hence, 
a  risk  evaluation  must  be  conducted  by 
someone  with  a  broad  perspective  of  the 
total  program  including  business,  technical 
and  programmatic  issues.  Additionally,  the 
impact  of  outside  influences  from  both  the 
government  and  industry  require  that 
multiple  participants  contribute  to  any 
successful  evaluation. 

Risk  management  is  an  organized  means 
of  identifying  and  measuring  risk  (risk 
assessment)  and  developing,  selecting,  and 
managing  options  (risk  analysis)  for  resolving 
(risk  handling)  these  risks.  Several  tools  are 
available  to  assist  the  program  office  in 
understanding  the  danger  signals  that  may 
indicate  the  program  is  off-track,  determining 
the  seriousness  of  the  problem,  and 
prioritizing  corrective  actions  as  necessary. 
With  the  addition  of  planning  we  call  the 
process  of  planning,  identifying,  quantifying, 
and  selecting  methods  to  handle  risk  the 
management  of  risk.  The  general  approach 


to  risk  management  is  presented  here,  while 
appropriate  chapters  incorporate  specific 
recommendations  for  managing  the  risks 
associated  with  each  systems  engineering 
activity. 

Risk  management  is  not  a  separate 
program  office  activity  assigned  to  a  risk 
management  branch,  but  rather  is  one  aspect 
of  a  sound  technical  management  program. 
Many  of  the  systems  engineering 
management  techniques  outlined  in  this  guide 
(e.g.,  technical  performance  measurement 
(TPM),  life  cycle  costing,  configuration 
management  (CM),  template,  trade-off 
analysis)  are  also  risk  management  tools  or 
techniques  used  for  risk  assessment  and  risk 
handling  methods. 

15.2  APPROACH 

It  is  important  that  a  risk  management 
strategy  be  established  early  in  a  program 
and  that  risk  be  continually  addressed 
throughout  the  system  life  cycle.  Risk 
management  includes  several  related  actions: 
1)  risk  planning,  2)  risk  assessment,  3)  risk 
analysis,  and  4)  risk  handling. 

15.2.1  Risk  Planning 

Risk  planning  is  the  process  to  force 
organized  purposeful  thought  to  the  subject 
of  eliminating,  minimizing,  or  containing  the 
effects  of  undesirable  occurrences.  The 
individual  assigned  this  task  should  instill  a 
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sense  of  ownership  with  the  functional 
managers  so  as  to  cause  their  continual 
examination  of  alternative  solutions 
(handling)  to  the  risk  areas  (problems).  As 
with  any  process  there  are  two  basic 
elements:  planning  and  execution,  which 
include  monitoring  and  control  techniques. 

15.2.2  Risk  Assessment 

Risk  assessment  is  the  process  of  examining 
a  situation  and  identifying  the  areas  of 
potential  risk.  The  first  step  in  risk 
management  is  to  identify  and  assess  all 
potential  risk  areas.  This  may  include  a 
survey  of  the  program,  customer,  and  users 
for  concerns  and  problems.  The 
thoroughness  with  which  this  identification  is 
accomplished  will  determine  the  effectiveness 
of  risk  management. 

Some  degree  of  risk  always  exists  in 
program,  technical,  test,  logistics,  production, 
and  engineering  areas.  Program  risks  include 
funding,  schedule,  contract  relationships,  and 
political  risks.  Technical  risks  may  involve 
the  risk  of  meeting  a  performance 
requirement  such  as  reliability,  probability  of 
first  weapon  hit,  maneuverability  or 
survivability,  but  may  also  involve  risks  in  the 
feasibility  of  a  design  concept  or  the  risks 
associated  with  using  state-of-the-art 
equipment  or  software.  Production  risk 
includes  concerns  over  quality,  rework, 
producibility,  packaging,  manufacturing,  lead 
times,  and  material  availability.  Engineering 
risks  associated  with  user  suitability  include 
reliability,  maintainability,  operability,  and 
trainability  concerns.  The  understanding  of 
risks  in  these  and  other  areas  evolves  over 


time.  Consequently,  the  risk  assessment  must 
continue  through  all  program  phases. 

The  methods  for  identifying  risk  are 
numerous.  Any  source  of  information  that 
allows  recognition  of  a  potential  problem  can 
be  used  for  risk  identification.  These 
include: 

a.  Systems  engineering  documentation 

b.  Life  cycle  cost  analysis 

c.  Schedule  analysis 

d.  Baseline  cost  estimates 

e.  Requirements  documents 

f.  Lessons-leamed  files 

g.  Trade  studies/analyses 

h.  Technical  performance  measurement 
(TPM)  planning/  analysis 

i.  Cost  Models 

j.  Scheduling  Models 

k.  Work  Breakdown  Structures  (WBS) 

DoD  4245.7-M  [1]  provides  a  structure, 
which  is  becoming  a  standard  within  DoD, 
for  identifying  technical  risk  areas  in  the 
transition  from  development  to  production. 
The  structure  is  geared  towards  development 
programs  but,  with  modifications,  could  be 
used  for  any  acquisition  program.  The 
structure,  shown  in  Figure  15-1,  identifies  a 
template  for  each  major  program  technical 
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PRODUCT 


Figure  15-1 

DoD  4245. 7 -M  Risk  Management  Templates 
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management  and  systems  engineering  activity. 

Each  template  identifies  potential  areas 
of  risk.  Overlaying  each  template  on  a 
program  allows  identification  of  mismatched 
areas,  which  are  then  identified  as  "at  risk." 
Having  used  all  applicable  templates,  the 
Program  Manager  will  have  created  a  'Svatch 
list"  of  production  transition  risk  areas  and 
can  prioritize  control  actions  ~  many  of 
which  will  be  the  responsibility  of  systems 
engineering.  DoD  4245.7-M  [1]  describes 
technical  methods  for  reducing  the  risk  in 
each  identified  area  (see  Figure  15-2). 

High  risk  areas  may  reflect  missing 
capabilities  in  the  PM’s  organization  or  in 
supporting  organizations.  They  may  also 
reflect  technical  difficulties  in  the  design  or 
development  process.  In  either  case, 
"management"  of  risk  involves  using  program 
management  assets  to  reduce  the  identified 
risks. 


One  approach  for  identifying  technical 
risk  areas  at  the  svstem  level  is  to  use  the 
guidance  provided  in  Reference  [2].  This 
document  outlines  a  rigorous  process  for 
identifying  specific  technical  risks  at  the 
functional  level  and  translating  this  detailed 
information  through  several  steps  into  a 
program  summary-level  risk  assessment,  as 
shown  in  Figure  15-3.  In  this  way,  risks  are 
identified  at  management  levels  appropriate 
to  monitoring  or  correcting  the  situation. 

Risk  monitoring  and  reporting  structures 
usually  identify  five  major  risk  categories: 
design  (performance),  test,  production,  cost, 
and  management.  These  categories  should 
be  examined  at  the  program,  subsystem,  and 
functional  levels.  Starting  from  the  bottom, 
data  are  developed  for  these  five  engineering 
indicators  in  order  to  rate  them  according  to 
"high,"  "medium,"  or  "low"  risk  categories  at 
the  subsystem  level.  Results  of  these 
assessments  should  be  summarized  for  each 


Figure  15-2 

Sample  Risk  Management  Template  (DoD  4275. 7-M) 
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Figure  15-3 

Technical  Risk  Identification  at  Appropriate 
Manage  sent  Levels 


subsystem  to  provide  a  system  overview,  as 
illustrated  in  Figure  15-4  for  a  torpedo 
development  program,  using  template  risk 
reduction  methods.  (Red-yellow-green 
notation  indicates  high-medium-low  risks 
respectively). 

A  risk  viewed  as  easily  manageable  by 
some  managers  may  be  considered  hard  to 
manage  by  less  experienced  or  less 
knowledgeable  managers.  Consequently,  the 
terms  "high,"  "medium,"  or  "low"  risk  are 
relative  terms.  Some  managers  may  be  risk 
averse  and  choose  to  avoid  recognized  risk 
at  all  reasonable  cost  Other  managers  may 
be  risk  seekers  and  actually  prefer  to  take  an 
approach  with  more  risk.  The  terms  "high," 
"medium,"  and  "low"  risk  may  change  with 
the  turnover  of  managers  and  their  superiors 


as  much  as  with  the  program  events. 

Another  approach  is  to  use  the  work 
breakdown  structure  (WBS)  as  the 
framework  for  risk  analysis  (refer  to  Chapter 
9  of  this  guide).  To  use  this  approach,  each 
element  at  level  3  of  the  WBS  is  further 
broken  down  to  the  fourth  or  fifth  level  and 
is  subjected  to  a  quantitative  risk  analysis 
procedure.  Items  at  system,  segment  or 
group,  or  subsystem  levels,  as  well  as 
management  items,  are  assessed  using 
attributes  such  as  maturity  and  complexity  of 
hardware  and  software  items  or  the 
dependency  of  the  item  on  existing  systems, 
facilities,  or  contractors  to  evaluate  their  risk 
levels.  Risk  may  be  identified  at  the 
summary  level  of  the  WBS  by  the  absence  of 
system  components,  such  as  training  devices. 
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Figure  15-4 

Variation  of  Risk  Identification  Products 
With  Management  Level 
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Lower  levels  of  the  WBS  are  typically  used 
for  allocation  of  risk  control  responsibilities. 

More  recently,  PMs  have  recognized 
that  it  is  the  systems  engineer  and  his  design 
methodology  that  decides  what  technology 
will  be  selected,  tolerances  assigned,  material 
finishes  required,  production  processes  called 
out,  and  tests  required  during  the  production 
process.  Each  of  these  examples  can  be 
recognized  as  effecting  the  quality  and  thus 
the  cost  of  the  item  in  production.  It  has 
been  recognized  that  the  quality  desired  in 
production  must  be  addressed  during  the 
design  phase  by  having  the  production 
engineers  work  with  the  design  engineers  to 
ensure  the  production  process  is  developed 
concurrently  with  the  actual  design  of  the 
weapon  system.  The  organization  that  does 
not  have  production  engineers  working  to 
concurrently  address  the  production  process 
with  the  design  will  most  likely  face  the 
following  risks:  lower  quality  and  higher 
rework,  a  much  higher  first  unit  cost,  a 
failure  to  reduce  cost  while  the  transition 
problems  are  being  resolved,  and  then  a 
recognition  of  cost  opportunity  lost  (by  trying 
to  work  a  production  process  that  was 
identified  by  design  but  is  found  to  be  less 
than  optimal  for  the  design  trying  to  be 
produced). 

In  house  design  reviews  are  an  excellent 
means  for  identifying  the  risks  being  taken 
through  the  various  technical  approaches  on 
a  program.  Additionally,  when  industry 
presents  their  design  reviews  to  the 
government  program  team,  MIL-STD-1521B 
[3]  requires  that  the  elements  of  technical 
risk  in  a  program  be  included  in  each  of  the 
program  reviews.  This  is  further  discussed  in 
Chapter  12  of  this  guide.  Risk  areas  and 


levels  should  be  identified  during  trade-off 
analysis  and  included  if  appropriate  in  the 
review  agenda.  Risk-handling  options  should 
be  addressed  at  about  the  WBS  level  3, 
during  the  program  review.  The  discussing 
of  system  and  program  risk  in  program 
reviews  is  an  essential  element  of  the 
technical  management  process. 

The  value  in  each  of  these  approaches  to 
risk  identification  lies  in  the  methodical 
nature  of  the  approach,  which  forces 
disciplined,  consistent  treatment  of  risk. 
However,  using  any  method  in  a  "cookbook" 
manner  may  cause  unique  risk  aspects  of  the 
program  to  be  overlooked.  Before  acting  on 
the  outcome  of  any  assessment,  the  Program 
Manager  must  review  the  strengths  and 
weaknesses  of  the  approach  and  insure  that 
other  factors  that  may  introduce  technical, 
schedule,  cost,  and  program  risk  have  not 
been  overlooked  as  a  result  of  the  method 
used. 

15.2.3  Risk  Analysis 

Risk  analysis  requires  conducting  an 
analysis  of  the  program,  as  well  as  any 
environmental  changes,  to  determine  the 
probability  of  events  and  the  consequences 
associated  with  the  potential  actions  that 
could  affect  the  program.  Many  tools  exist 
to  aid  in  the  analysis,  such  as  schedule 
network  models  and  life  cycle  cost  models. 
The  purpose  of  risk  analysis  is  to  discover 
the  cause,  effects,  and  magnitude  of  the  risk 
perceived,  and  to  develop  and  examine 
alternative  options. 

Describing  and  quantifying  a  specific  risk 
and  the  magnitude  of  that  risk  usually 
requires  some  modeling.  Typical  tools  for 
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use  in  risk  analysis  are: 

a.  Schedule  Network  Model  -  Venture 
Evaluation  Review  Technique  (VERT),  or 
Program  Evaluation  Review  Technique 
(PERT)  (see  References  (4]  and  (5])  with 
dependent  activities  networked  for  analysis 
of  the  impact  to  any  activity. 

b.  Life  Cycle  Cost  (LCC)  Model  -  tailored 
for  the  phase  of  development  and  type  of 
system  (LCC  models  are  discussed  in 
Chapter  17  of  this  guide). 

c.  Quick  Reaction  Rate/  Quantity  Cost 
Impact  Model  -  such  as  Defense  Production 
Engineer  Support  Office  model. 

The  first  two  models  are  used  to  examine 
schedule  and  program  cost  risk  as  various 
options  for  procurement  strategy,  logistics 
maintenance  levels,  and  contractor  support 
are  considered.  The  quick  reaction  model  is 
used  when  program  funds  are  challenged,  an 
action  that  happens  frequently  during  the 
budget  cycle.  These  challenges  are  most 
prevalent  just  prior  to  production  and  can 
severely  affect  the  program’s  executability. 
The  quick  reaction  model  also  permits 
examination  of  probable  cost  impacts  for 
determining  required  budget  allocation  in 
program  planning  of  various  options,  such  as 
use  of  contractor  maintenance,  or  joint 
software  support  facilities. 

Network  models  identify  the  key 
interactions  of  the  major  activities 
throughout  a  program  or  a  phase  of  the 
program.  Most  modern-day  networks  permit 
the  probability  of  each  activity  to  be  varied 
so  that  the  sensitivity  of  key  parameters 
upon  models  of  total  cost  or  schedule  can  be 


examined.  This  information  is  used  in 
determining  resource  allocation  priorities. 

For  every  area  of  risk,  there  is  a  method 
of  analysis  to  determine  the  logical 
relationships  of  cause  and  effect.  Some  of 
the  most  frequent  causes  of  program  impact 
are  production  rate  and  quantity  change, 
inadequate  planning  for  production,  and  a 
failure  to  adequately  prepare  for  the  system 
support  requirements.  Test  and  evaluation 
is  usually  considered  a  risk  reduction 
measure  in  that  T&E  identifies  items, 
subsystems  or  systems  that  have  identifiable 
shortfalls  relative  to  the  requirement. 

A  product  of  risk  analysis  is  a  " watch  list". 
This  is  the  identification  of  consequences 
that  are  likely  to  occur  and  the  indicators  of 
the  start  of  the  problem.  An  example  of 
this  is  the  cost  risk  of  production  due  to  an 
immature  technical  data  package.  When 
production  starts  before  the  technical  data 
package  has  been  adequately  engineered  for 
producibility,  the  first  unit  cost  is  higher  than 
planned  with  a  "steeper  than  planned”  learning 
curve.  This  is  the  result  of  excessive 
producibility  changes  needed  to  achieve  the 
planned  production  cost.  Therefore,  when 
the  first  article  cost  is  estimated  during 
production,  it  is  time  to  reassess  the 
technical  data  package  for  need  to  put  more 
than  planned  emphasis  on  producibility 
engineering  efforts.  A  typical  watch  list  is 
structured  to  show  the  trigger  event  or  item 
(for  example,  long  lead  items  delayed),  the 
related  area  of  impact  (production  schedule) 
and  iater,  as  they  are  developed,  the  risk 
handling  actions  taken  to  avoid/  minimize 
the  potential  for  or  impact  from  that  event 
(such  as  ensuring  early  identification  of  long 
lead  items  or  placing  contractor  emphasis  on 
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early  delivery). 

The  watch  list  is  periodically  reevaluated 
and  items  are  added,  modified,  or  deleted  as 
appropriate.  Should  the  trigger  events  on 
the  watch  list  occur  during  a  program,  there 
would  he  immediate  cause  for  impact 
assessments  to  be  updated  and  risk  handling 
methods  to  be  selected. 

15.2.4  Risk  Handling 

Risk  handling  includes  techniques  and 
methods  developed  to  reduce  or  control  the 
risk.  There  is  no  risk  management  if  there 
are  no  provisions  for  handling  the  identified 
and  quantified  risk. 

The  techniques  for  reducing  or 
controlling  risk  fall  into  the  following 
categories:  1)  avoidance,  2)  prevention 
(control),  3)  assumption  (retention),  4) 
transfer,  and  5)  knowledge  and  research. 

To  avoid  risk  is  to  avoid  the  potential 
failure  consequence  and/or  its  probability. 
In  program  management,  risk  avoidance  may 
be  reflected  in  the  system  concept  selection 
and  contractor  source  selection.  Not  every 
risk  can  be  wholly  avoided.  An  action  that 
avoids  one  risk  may  simply  transfer  that  risk 
to  another  area.  The  process  of  source 
selection  allows  the  PM  to  identify'  the 
shortcomings  of  competitive  sources  so  as  to 
avoid  sources  having  unacceptable  risk. 

Risk  control  is  the  process  of  continually 
sensing  the  condition  of  a  program  and 
developing  options  and  fall  back  positions  to 
permit  alternative  lower  risk  solutions. 
Technical  performance  measurement  (TPM) 


and  cost/schedule  control  system  criteria 
(C/SCSC)  are  the  best-known  sensors.  The 
process  of  developing  alternative  sources  for 
production,  parallel  development  for  a 
critical  research  and  design  component,  or 
getting  priority  for  critical  materials  are  all 
examples  of  risk  control  options.  The 
requirements  (in  time  and  money)  for 
development  of  these  options  determine  the 
required  sensitivity  of  the  monitoring  tools 
selected  by  the  program  office.  A  TPM  or 
cost  performance  measurement  (CPM) 
report  giving  90  days  notice  of  an  impending 
failure  is  not  helpful  if  the  products  of 
parallel  development  will  take  six  months. 

Risk  assumption  is  an  acknowledgement 
of  the  existence  of  the  risk  but  a  decision  to 
accept  the  consequences  if  failure  occurs. 
Most  acquisition  programs  and  all 
development  programs  must  assume  some 
risks.  Most  acquisition  programs  and  all 
development  programs  must  assume  some 
risks.  Identification,  analysis,  and  selection 
of  handling  techniques  allow  the  PM  to 
assume  the  "right"  risks,  such  as  those  with 
either  low  probability,  low  consequences  or 
both.  Those  that  are  too  hazardous  to 
assume  may  be,  at  least  partially,  transferred 
to  the  contractor  at  appropriate  premium 
cost. 

Options  for  risk  transfer  from  the 
government  PM  to  the  contractor  include 
product  performance  incentives,  warranties, 
cost  incentives,  and  fixed  price  contracts. 
These  are  agreements  with  the  contractor 
that  the  consequent  "costs"  of  failure  will  be 
assumed  by  the  contractor  at  an  agreed 
upon  price  which  may  be  in  terms  of  profit 
dollars,  product  performance  modifications, 
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or  schedule  changes. 

Knowledge  and  research  as  a  method  for 
risk  handling  is  a  continuing  process  that 
enables  the  participants  to  perform  risk 
reductions  through  both  probability  and 
consequence  modification  such  as: 

a.  Early  initiation  of  development  activities 

b.  Implementation  of  extensive  testing 

c.  Development  of  simulations  to  predict 
performance. 

Risk  managers  draw  on  methods, 
techniques,  and  tools  available  in  decision 
analysis  disciplines  to  determine  the 
preferred  course  of  action.  Various  decision 
analysis  tools,  such  as  trade-off  analyses  (see 
Chapter  8  of  this  guide),  are  described 
elsewhere  in  this  guide.  In  addition,  the 
principles  of  utility  analysis  and  decision 
making  under  uncertainty  (see  Reference 
[4J)  provide  useful  tools  in  many 
applications.  One  special  risk  abatement 
tool  developed  by  the  Army  is  Total  Risk 
Assessing  Cost  Estimates  (TRACE).  The 
principle  of  TRACE  is  to  consider  the  cost 
of  risk  for  a  command  (including  all  its 
ongoing  programs)  and  to  pool  resources 
against  the  total  assessed  risk  for 
apportionment  according  to  needs  over  time. 
Since  TRACE  funds  are  usually  a  calculation 
based  on  probability  and  consequences  of 
known  or  perceived  problems,  they  do  not 
provide  for  the  risks  identified  as  "unknown- 
unknowns"  such  as  a  labor  strike  at  a 
production  facility,  fires,  acts  of  god  or  other 
non  perceived  problems.  Since  the  funds  set 
aside  would  always  be  only  a  sub-set  of  the 
total  unknowns  (e.g.,  both  the  known- 


unknowns  such  as  inflation  variance, 
schedule  variance,  or  performance  shortfalls, 
and  unknown-unknowns)  they  would  never 
cover  all  risks.  This  would  require  that  the 
funds  set  aside  for  TRACE  were  indeed 
"risk"  funds  and  not  management  reserve  or 
funds  for  highly  probable  unknowns. 

1SJ  AVOIDING  COMMON  TRAPS 

Identified  risks  must  be  managed  so  as  to 
mitigate,  to  the  extent  assets  allow,  their 
potential  to  adversely  affect  the  program. 
All  levels  of  management  must  be  sensitive 
to  hidden  "traps"  that  may  induce  a  false 
sense  of  security.  If  properly  interpreted, 
these  signals  really  indicate  a  developing 
problem  in  a  known  area  of  risk.  NAVSO 
P  6071  Id]  offers  guidelines  to  the  PM  in 
implementing  DoD  4245.7-M  [1]  at  the 
working  level.  The  document  follow's  the 
same  template  structure  as  DoD  4245.7-M 
{!].  Each  trap  is  accompanied  by  several 
"warning  signs"  that  show  an  approaching 
problem  and  the  likely  ’consequence"  of 
failing  to  treat  the  problem  at  its  inception. 
The  document  also  suggests  "escapes,"  or 
ways  to  avoid  these  common  traps,  and 
notes  attendant  "benefits"  from  exercising 
positive  control  over  that  technical  risk  area. 
An  example  is  shown  in  Figure  15-5. 

The  ability  to  turn  traps  into  advantages 
suggests  that  much  of  the  technical  risk  in  a 
program  can  be  actively  mitigated,  not 
merely  watched  and  resolved  after  a 
problem  occurs.  In  some  instances  it  may 
pay  to  watch  and  wait.  If  the  probability 
that  a  certain  problem  will  arise  is  low  or  if 
the  cost  exceeds  the  benefits  of  "fixing"  the 
problem  before  it  happens,  a  do-nothing 
alternative  may  be  advisable.  Effective  risk 
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Figure  15-5 

Sample  Risk  Reduction  Approach  ("Best  Practices") 
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management  makes  selection  of  the 
do-nothing  alternative  a  conscious  decision 
rather  than  an  oversight  and  should  trigger 
an  appropriate  addition  to  the  watch  list  and 
perhaps  to  formal  TPM  reporting 
requirements. 

"Best  Practices"  acknowledges  that  all  of 
the  traps  have  not  been  identified  for  each 
template.  The  traps  are  intended  to.  be 
suggestive,  and  other  potential  problems 
should  be  examined  as  they  arise.  It  is  also 

important  to  recognize  that  the  sources  and 
types  of  risk  evolve  over  time.  Particularly 
during  the  transition  from  development  to 
production,  the  nature  of  potential  problems 
shifts.  Risks  may  take  a  long  time  to  mature 
into  problems,  so  attention  may  be  properly 
focused  in  advance  of  the  current  acquisition 
phase. 

15.4  RISK  MONITORING 

Risk  items  are  monitored  continually  by 
systems  engineers  and  risk  reduction  actions 
are  recommended.  Inherent  in  the 
monitoring  of  technical  performance  design 
risk  is  evaluation  of  predicted  performance 
against  specified  requirements.  Appropriate 
performance  parameters  for  risk  monitoring 
are  established  at  the  top  level,  together 
with  their  contributors  (or  allocations)  at 
lower  levels.  Properly  managed  systems 
engineering  ensures  that  the  risks  associated 
with  each  design  decision  are  identified  and 
treated  in  risk  assessment  updates:  how  to 
recognize  the  risk  if  the  potential  problem 
should  occur  (e.g.,  higher  failures  when 
stress  is  over  60  foot-pounds)  and  what 
actions  should  be  taken  if  the  problem  is 
due  to  the  potential  risk  area  (e.g.,  use 


substitute  component  X27-46).  In  addition, 
for  the  total  system  concept,  the  relationship 
between  the  systems  engineer  and  the 
project  control  chief  should  be  indicated, 
showing  the  contribution  of  each  to  the 
selections  made  in  the  decision  process. 

The  risk  item  performance  parameter  is 
monitored  as  part  of  TPM,  and  action  is 
taken  dependent  on  its  effect  on  system 
level  performance.  Costs  and  schedules  are 
monitored  by  reports  generated  to  satisfy 
C/SCSC  (refer  to  MIL-STD-881A  [7]). 

15.5  DOCUMENTATION 

Program  risk  documentation 
requirements  imposed  on  contractors  vary 
and  are  highly  dependent  upon  the  nature  of 
the  program  contract  type  and  the  program 
phase.  The  following  contractor 
documentation  may  be  required: 

a.  Risk  Management  Program  Plan 

b.  Risk  Sensitivity  Analysis 

c.  Risk  Handling  Plans 

d.  Risk  Reduction  Reports. 

15.5.1  Risk  Management  Program  Plan 

A  Risk  Management  Program  Plan  may 
be  initiated  by  each  contractor  in  the 
Concept  Exploration/  Definition  (C/E)  phase 
and  updated  at  each  phase  of  the 
development  cycle  to  reflect  a  degree  of 
detail  and  approach  consistent  with  phase 
objectives.  The  program  office  should 
consider  the  contractor’s  approach  to  risk 
management  in  its  evaluation  of  contractor 
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proposals. 

A  government-approved  Risk 
Management  Program  Plan  may  be  required 
of  the  prime  contractor  before  the  System 
Design  Review  (SDR).  If  no  risk 
management  plan  is  required,  the  industry 
prepared  Systems  Engineering  Management 
Plan  (SEMP)  should  describe  the  risk 
analysis  approach  being  used  for  the 
program. 

The  Risk  Management  Program  Plan 
would  describe  programmatic  aspects  of  risk 
planning,  risk  identification,  risk  assessment, 
risk  reduction,  and  risk  management 
functions  to  be  performed  by  the  contractor. 
The  risk  management  plan  should  relate  the 
contractor’s  approach  for  handling  risk  to 
the  options  (e.g.,  avoidance,  prevention, 
transfer)  discussed  in  Section  15.2.3.  A 
suggested  outline  for  the  plan  is  given  in 
Figure  15-6. 

The  plan  should  be  tailored  to  reflect 
program  concerns.  It  should  describe  a  risk 
reduction  program  with  minimum  cost  and 
schedule  impact,  using  a  minimum  number 
of  personnel.  It  should  describe  a  simple 
but  complete  method  of  surveying  individual 
potential  risks  and  identifying  the  degree  or 
level  of  risk  at  each  system  level.  The  plan 
should  also  describe  how  an  iterative  risk 
assessment  process  is  applied  at  all  WBS  or 
Contract  Work  Breakdown  Structure 
(CWBS)  levels  for  each  previously  identified 
risk  as  the  design  progresses. 

The  plan  should  describe  the  role  of  risk 
assessment  in  design  reviews,  technical 
performance  monitoring,  and  the  change 
control  processes.  It  should  describe  the 


available  methods  of  risk  reduction, 
monitoring,  and  management  for  each 
defined  and  assessed  level  of  risk. 

The  plan  should  provide  the  means  for 
ensuring  that  risk  assessment  of  associate 
contractors  and  subcontractors  is  consistent 
and  compatible  with  the  prime  contractor’s 
methods. 

The  Risk  Management  Program  Plan 
should  require  that  a  separate  Risk  Handling 
Plan  be  prepared  for  each  high  risk  item, 
identifying  the  timing  for  its  development 
and  assigning  originator  and  review 
responsibility.  The  plan  should  also  require 
that  Risk  Reduction  Reports  be  prepared 
for  each  item  classified  as  medium  or  high 
risk.  It  should  describe  the  role  of  the  plan 
in  the  program,  how  it  is  to  be  implemented, 
its  frequency,  its  relationship  to  systems 
engi'  eering,  and  how  the  risk  program  is  to 
be  managed.  The  approved  data  item 
description  for  contractor-generated  Risk 
Management  Plans  is  UDI-A-23862  [8]. 

15.5.2  Risk  Sensitivity  Analysis 

The  Risk  Sensitivity  Analysis  presents  the 
program’s  sensitivity  to  risk  in  terms  of 
schedule  and  cost.  It  examines  the  impact 
of  each  risk  element  down  to  level  3  of  the 
WBS  on  the  overall  program  cost  and 
schedule.  The  analysis  relates  directly  to  the 
Risk  Management  Program  Plan  and  first 
summarizes  the  potential  cost  and  schedule 
if  no  risk  reduction  actions  are  taken.  This 
summary  is  followed  by  a  detailed 
examination  of  each  WBS  area  to  determine 
item  should  be  prepared  by  the  contractor  in 
accordance  with  the  SEMP  and  submitted  to 
the  government  for  review  and  approval 
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Figure  15-6 

Sample  Risk  Management  Program  Plan  Outline 


I. 

Int  roduc t ion 

1 .  Overview 

2.  Applicable  Documents/Definitions 

3.  Management  0rgani2a t ion/Respons ibi 1 i t ies 

4.  Scheduled  Milestones/Reviews 

II. 

Identification  and  Assessment 

5.  Survey  and  Identification 

6.  Risk  Assessment  Models 

7.  Flow/Level  Assessment /Treatment 

8.  System  Hierarchy  and  Risk  Tree 

Ill . 

Analysis  and  Reduction 

9.  Reduction  Methods 

10.  Analysis  Methods 

11.  Risk  Abatement  Plan 

12.  Prototyping/Simulations/Tests 

IV. 

Appendices 

A.  Survey  Form 

B.  Report  Format  and  Content 

C.  Assessment  Tables/Graphs 

D.  Plan  Format  and  Content 

prior  to  implementation.  These  plans  could 
be  developed  at  the  end  of  each  phase  for 
implementation  in  the  next  phase  or  upon 
identification  of  a  high  risk  item  by  the 
contractor.  Progress  to  plan  should  be 
reviewed  at  program  technical  reviews. 
Suggested  plan  contents  include: 

a.  Statement  and  assessment  of 
risk/problem 

b.  Consequences  of  failure 

c.  Alternatives  considered  with  risk  and 
cost  of  each 

d.  Recommended  risk  reduction/abatement 
method 


e.  Implementation  impact  statement  (cost/ 
schedule/  technical) 

f.  Responsible  organization  and  personnel 

g.  Implementation  start  date  and  key 
milestone  schedule 

h.  Criteria  for  closure  of  this  risk  activity 

i.  Decision  points 

j.  Recommended  back-up  developments 
and  tests  including  cost. 

Risk  Handling  Plans  should  be  limited  to 
the  highest  risk  items  if  possible,  as  they  are 
manpower  and/or  cost  intensive.  They 
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should  also  be  tailored  to  fit  the  program 
and  to  satisfy  government  requirements. 

15.5.4  Risk  Reduction  Report 

Contractors  should  submit  a  Risk 
Reduction  Report  to  the  government  for 
each  medium  or  high  risk  item  (not  less  than 
one  report  for  each  level  3  WBS  element). 


The  report  should  be  updated  periodically, 
as  specified  by  the  government,  to  describe 
the  status  of  risk  reduction  programs  being 
implemented.  The  updates  should  be 
expected  to  coincide  with  technical  reviews, 
and  to  proceed  government  in-house  reviews 
such  as  those  scheduled  with  higher 
headquarters. 
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CHAPTER  16 


MODIFICATION  MANAGEMENT 


16.1  INTRODUCTION 

Department  of  Defense  (DoD)  weapon 
systems  undergo  modifications  throughout 
their  life  cycles.  For  example,  1)  a  typical 
tactical  aircraft  goes  through  four  to  five 
major  modifications  after  deployment,  2)  the 
M-60  tank  has  undergone  35  changes  since 
initial  production,  and  3)  the  UH-1 
helicopter  has  undergone  45  changes.  What 
this  means  is  that  1)  a  substantial  proportion 
of  the  DoD  budget  is  used  to  modify  existing 
systems  vice  developing  new  systemsj  and  2) 
the  management  of  these  modifications  or 
changes  to  existing  systems  becomes  more 
critical  and  requires  special  emphasis  on 
integration,  implementation,  and  baseline 
control.  Modifications  are  used  to  correct 
system  deficiencies  to  provide  increased 
performance,  counter  new  threats,  lower  life 
cycle  costs,  extend  the  system’s  useful  life,  or 
remove  obsolete  capabilities. 

Planning  for  future  modifications  starts  in 
development.  Design  decisions  made  in 
development  will  affect  the  flexibility  of  the 
system  to  incorporate  changes  throughout  its 
life  cycle.  Once  the  system  is  produced,  the 
decisions  are  costly  to  reverse.  Especially 
during  the  transition  from  development  to 
production,  ease  of  future  modification  must 
be  considered  in  conjunction  with 
performance,  cost,  and  manufacturing  and 
producibility  considerations.  For  example,  it 
may  be  easier  or  less  expensive  initially  to 
weld  a  seam  (the  welded  seam  may  reduce 


weight,  thereby  enhancing  performance); 
whereas,  if  modification  considerations  were 
evaluated,  a  bolted  seam  might  be  more  cost 
effective  over  the  system’s  life  cycle,  or 
might  simplify  a  future  upgrade  by  allowing 
a  modification  to  be  accomplished  in  the 
field. 

Modifications  sometimes  occur  after 
production  has  started  but,  due  to  the  length 
of  the  Production  phase,  it  may  be  possible 
to  incorporate  the  modification  into  some  of 
the  items  during  their  assembly/  fabrication 
instead  of  waiting  until  after  they  have  been 
"produced."  Incorporating  changes  into 
produced  items  is  complex  since  the 
produced  items  may  be  deployed  in  many 
locations  with  multiple  agencies  responsible 
for  their  operation,  support,  maintenance, 
and  repair.  Managing  this  aspect  of 
modifications  tests  the  best  management 
talents  and  requires  considerable  effort  to 
realize  successfully. 

During  production,  successful 
modification  programs  are  dependent  on  a 
highly  disciplined  configuration  management 
(CM)  system.  This  is  especially  true  when 
large  numbers  of  items  are  built  over 
prolonged  periods  and  are  widely  deployed 
(e.g.,  the  M-60  tank  or  the  F-4  aircraft). 
The  ability  to  make  even  minor  corrections 
at  remote  facilities  is  often  limited.  The  use 
of  a  CM  system  is  therefore  mandatory 
throughout  the  life  of  the  system.  Adequate 
communication  channels  are  essential  for 
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transferring  modification  data  from  the 
organization  installing  the  modification  to  the 
CM  center,  so  that  up-to-date  configuration 
status  accounting  records  can  be  maintained. 

There  is  a  clear  distinction  between 
post-production  modifications  and 
engineering  change  proposals  (ECPs).  When 
systems  receive  changes  during  production, 
only  those  systems  that  have  been  fielded 
can  be  modified  through  the  product 
improvement  program.  For  example,  Army 
systems  may  not  receive  product 
improvements  until  they  are  type  classified 
standard  and  accepted  into  the  inventory. 
Those  systems  still  in  production  must  be 
modified  by  ECPs  using  production  funds  or 
have  modification  kits  installed  after  they 
have  been  delivered  to  the  user.  Research, 
Development,  Test  and  Evaluation 
(RDT&E)  funds  are  used  to  develop 
modifications  that  increase  the  operational 
envelope  of  a  system;  Operation  and 
Support  (O&S)  funds  are  applied  to 
modification  kits.  When  system 
improvements  are  very  large,  such  as  the 
CH-47D  upgrade,  they  are  budgeted  and 
funded  as  if  they  were  new  development 
efforts.  Currently,  the  cost  of 
post-development  software  improvements  are 
paid  for  with  O&S  and  RDT&E  funds. 

If  a  modification  must  be  incorporated 
into  a  number  of  different  systems  (e.g., 
installation  of  the  cruise  missile  on  ships, 
aircraft,  and  missile  carriers),  this  multiplies 
the  complexity  of  the  modification.  Tailored 
installation  instructions  and  kit  interface 
components  are  required  for  each 
application.  Communications  with  agencies 
and  users  and  modification  management 
problems  also  increase  dramatically. 


16.2  APPROACH 

Modification  to  an  existing  hardware  or 
a  software  configuration  item  (Cl)  is  a 
change  to  its  form,  fit,  or  function  that 
revises  its  physical  or  functional  character¬ 
istics.  The  general  phasing  of  modifications 
in  the  acquisition  process  is  shown  in  Figure 
16-1. 

Modifications  are  classified  in  many  ways. 
No  matter  how  each  Service  component  may 
classify  modifications,  there  are  two  major 
categories  modifications  fall  into:  1)  product 
improvements  and  2)  preplanned  product 
improvements. 

Product  Improvement  (PI)  -  Pis  are 
applied  to  already  fielded  systems.  Pis  are 
undertaken  in  response  to  con-  gressional 
mandates,  cost  reduction  efforts,  reliability/ 
maintainability/  availability  improvements, 
rationalization,  standardization, 
interoperability  considerations,  or  safety- 
factors.  CIs  are  also  undertaken  to  improve 
operational  capabilities.  In  addition,  Pis 
may  be  used  to  correct  system  problems 
discovered  after  fielding,  although  less  than 
10  percent  of  Pis  are  needed  for  this 
purpose. 

Pre-Planned  Product  Improvement  (P*I) 

-  Recent  initiatives  to  improve  the  DoD 
acquisition  process  have  included  P*I.  This 
is  an  attempt  to  field  low  cost,  low  risk 
systems  with  preplanned  design  modifications 
keyed  to  foreseeable  technological 
breakthroughs  and  expected  changes  in  user 
needs.  PJI  is  a  systematic  and  orderly 
acquisition  strategy  to  facilitate  evolutionary, 
cost  effective  upgrading  (growth)  of  a 
system’s  capabilities  throughout  the  life  cycle. 


Figure  16-1 

Time  Phasing  of  Product  Improvement  Modifications 
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The  commercial  aircraft  industry  uses  this 
concept  when  planning  families  of  aircraft 
from  the  same  baseline  design.  Further 
detail  on  P-*I  is  given  in  Section  16.4. 

163  IMPLEMENTATION 

The  DoDD  5000.1  [1]  acquisition  cycle 
can  be  thought  of  as  composed  of  the 
following  Five  steps  for  modification 
programs:  1)  need,  2)  design,  3)  test,  4) 
manufacture,  and  5)  operate  and  support. 
The  need  and  test  steps  are  like  those  for 
other  programs  and  require  no  additional 
explanation  here. 

During  the  design  step,  the  modification 
and  associated  modification  kit  are  designed. 
The  modification  kit  is  the  collection  of 
hardware,  software,  data,  and  instructions 
that  incorporates  the  modification  into  the 
existing  system.  The  modification  should  be 
designed  so  that  it  can  be  incorporated  in  a 
produced  item  without  degrading  its 
performance,  if  possible.  Integrated  logistic 
support  (ILS)  planning  for  the  modified 
system  must  be  done  during  development. 

The  manufacture  step  involves 
manufacturing  and  assembling  the 
modification  kit.  The  kit  includes  not  only 
the  hardware  and  software,  but  also  the  data 
required  to  install  the  modification  and  to 
operate  and  support  the  modified  system. 
The  kit  may  also  contain  unique  tools, 
spares,  and  other  items  necessary  to 
incorporate  and  support  the  change.  Kit 
production  and  delivery  schedules  must  be 
fully  coordinated  with  the  installation 
schedule. 

Implementation  of  the  installation 
schedule  becomes  more  complex  as  the 


number  of  systems  that  will  receive  the 
modification  increases  and  as  the  number  of 
modifications  to  an  existing  system  increases. 
Non-standard  configurations  are  a  " known 
unknown "  and  should  be  considered  in 
modification  program  planning.  Successful 
modification  programs  proceed  on  the 
assumption  that  each  system’s  configuration 
may  vary  from  that  documented.  One 
solution  (although  expensive)  is  to  audit  each 
fielded  system  and  "upgrade"  the 
configuration,  as  necessary,  to  a  standard 
baseline  before  proceeding  with  the 
modification. 

The  modification  PM  is  responsible  for 
the  design  and  development  of  the 
modification.  Together  with  the  systems 
engineer,  the  modification  PM  assures  the 
integrity  of  the  existing  system  during  and 
after  modification.  The  systems  engineer 
will  assure  that  the  developers,  supporters, 
and  users  will  communicate  closely  and 
continuously  in  order  to  promote  a 
successful  modification  program. 
Communications  between  the  user  and  the 
developer  can  enhance  the  ability  of  the 
modification  to  meet  user  needs.  It  can  also 
eliminate  costly  rework  to  the  system  or 
modification  kit  and  reduce  the  likelihood  of 
adverse  performance  of  the  modified  system. 
The  modification  installation  schedule  should 
be  coordinated  with  the  user  to  ensure  that 
the  systems  are  available  for  making  the 
modification  and  that  trained  personnel  are 
available  to  operate,  maintain,  and  support 
the  modified  system.  Effective 
communication  channels  will  not  guarantee 
successful  modification  programs,  but  they 
will  facilitate  progress  toward  a  successful 
program.  Modification  requirements  must 
be  justified,  costed,  documented,  funded,  and 
accepted  by  the  user  community. 
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Planning  and  budgeting  for  the 
modification  is  similar  to  that  for  developing 
a  new  system.  The  PM  must  consider  any 
required  production  tooling,  test  equipment, 
support  equipment,  simulators,  and  trainers; 
as  well  as,  the  documentation,  training,  and 
operational  readiness  validation  for  the 
system  to  be  modified.  Of  particular 
importance  is  compatibility  between  different 
modifications,  especially  when  incorporated 
separately.  Problems  such  as 
electromagnetic  interference  could  result 
when  the  total  system  is  implemented, 
negating  the  use  of  one  or  more  of  the 
improvements.  Systems  engineers  should 
analyze  all  proposed  modifications  and 
recommend  or  disapprove  implementation 
to  the  appropriate  Configuration  Control 
Board  (CCB)  or  decision  authority. 

16,4  SPECIAL  APPLICATIONS: 
PRE-PLANNED  PRODUCT 
IMPROVEMENTS 

There  is  no  single  DoD  specified 
approach  to  modification  management. 
However,  the  Deputy  Secretary  of  Defense, 
in  DoDD  5000.1  [I],  directed  the 

implementation  of  P3I  in  major  DoD 
programs.  The  primary  objectives  of  P3I 
are: 

a.  Introduction  of  higher  technological 
performance  during  the  system’s  lifetime 
through  more  rapid  fielding  of  technological 
advances 

b.  Shortening  of  acquisition  and 
deployment  times 

c.  Extension  of  the  system’s  useful  life 
(before  obsolescence) 


a.  Reduction  of  system  technical, 
schedule,  and  cost  risk 

e.  Reduction  of  requirements  for  major 
system  new  starts 

f.  Higher  operational  readiness  during 
the  system’s  lifetime. 

The  P3I  concept  cannot  be  applied  to  all 
new  system  developments  but  should  be 
considered  when: 

a.  A  near-term  need  exists  to  build  a 
system  with  current  technology 

b.  There  is  a  high  risk  that  current 
technology  will  not  meet  a  projected  future 
threat  and  a  low  risk  that  future  technology 
will  not  meet  such  a  threat 

c.  The  system  can  be  designed  to 
incorporate  planned  technology  development 
(The  most  critical  element  is  the  ability  to 
modulate  the  system  to  minimize  integration 
and  retrofit  problems) 

d.  P3!  can  be  an  effective  means  of 
meeting  overall  long  term  program  objectives 
(based  on  threat,  development  risk,  and  total 
life  cycle  cost).  It  may  not  be  cost  effective 
for  low  cost,  low  technology  systems 

e.  A  long  term  military  need  exists  for 
the  system  (P^I  can  shorten  the  development 
time  for  the  basic  system;  however, 
evolutionary  changes  will  normally  lengthen 
the  total  development  period) 

f.  The  service,  DoD,  and  Congress 
demonstrate  a  commitment  to  acquire  the 
system  under  the  P3!  concept,  including 
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acceptance  of  initially  higher  costs. 

P-*I  represents  a  separate  and  specific 
acquisition  strategy  to  acquire  clearly  stated 
requirements  on  an  incremental  basis,  it  is 
not  a  process  where  the  PM  or  the  user 
attempts  to  guess  future  requirements  and 
configure  the  basic  system  to  accommodate 
these  guesses.  P^I  is  evolutionary 
development  or  incrementalism.  The 
mission  area  analysis  (MAA)  process  results 
in  the  identification  of  a  battlefield 
deficiency.  The  PM  translates  this  deficiency 
into  a  specified  materiel  solution.  When  the 
specified  solution  is  evaluated  as  " high  risk ", 
near  term,  reduced  risk  technologies  are 
selected  for  the  basic  system  in  favor  c* 
parallel  development  of  the  deferred 
technology.  Growth  provisions  are  then 
designed  into  the  basic  system  to 
accommodate  future  application  of  the 
higher  technology.  This  further  requirement 
must  be  clearly  stated  in  the  requirements 
document  and  validated  by  the  user. 

An  example  of  the  type  of  requirement 
that  can  be  accommodated  through  PJI  is  as 
follows:  "Fire  and  forget  preferred,  but 
required  not  later  than  6  years  after  initial 
operating  capability,"  or  "35  kilometer  range 
required  not  later  than  1995."  The  specific 
incremental  capability  improvement  must  be 
known  in  order  to  design  appropriate  growth 
provisions  into  the  system.  It  is  also  needed 
to  support  the  parallel  development  effort 
and  the  continuation  of  RDT&E  funds  past 
the  production  decision  of  the  basic  system. 
When  this  concept  is  not  followed  or  is 
poorly  understood,  then  goldplating  results 
and  sunk  costs  are  built  into  systems, 
because  of  poor  guesses.  When  the 
deferred  capability  is  applied  to  the  basic 
system,  it  is  accomplished  under  the  rules  of 


standard  product  improvement.  In  addition, 
during  the  upgrade  process,  growth 
provisions  are  also  installed  to  accommodate 
the  next  incremental  upgrade  as  necessary  in 
response  to  changes  in  the  validated  new 
future  requirement.  This  process  continues 
until  the  basic  design  constraints  dictate  a 
replacement  system. 

Figure  16-2  provides  suggested  criteria 
for  choosing  P-*I  over  a  new  start  or  a 
product  improvement.  It  should  be  noted 
that  no  single,  standardized  approach  can  be 
used  for  every  system  development. 

The  decision  to  use  PJI  should  be  made 
as  early  in  the  Concept  Exploration/ 
Definition  (C/E)  phase  as  possible  and  no 
later  than  Milestone  I.  In  Grder  to  make 
P-*I  effective,  the  design  strategy  should 
include:  1)  modular  design,  2)  a  careful 
architectural  interface  system,  and  3) 
provisions  for  space,  weight,  cooling,  and 
power,  for  example.  A  development  process 
must  be  established  to  communicate  system 
growth  requirements  and  identify  new 
technological  opportunities.  Implementation 
begins  shortly  after  the  design  strategy  is 
developed,  so  that  PJI  is  incorporated  into 
the  acquisition  strategy  at  the  outset.  The 
P*I  acquisition  strategy  should  be 
communicated  to  industry  early  in  the 
program,  and  industry  should  be  included  in 
the  process  of  developing  the  strategy. 

The  initial  request  for  proposal  (RFP) 
might  specify  additional  platform  capacity, 
volume,  or  other  characteristics  not  needed 
at  the  time  of  the  initial  configuration  freeze 
but  anticipated  for  later  use.  Further,  it 
might  specify  potentially  valuable  modular 
design  features  in  certain  areas.  The 
flexibility  of  the  design  to  incorporate  change 
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Figure  16-2 
Criteria  for  P 1  I 
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would  then  be  a  prominent  factor  in  design 
evaluation.  The  ability  of  the  offerors  to 
propose  and  conduct  a  P-*I  program  would 
also  be  part  of  the  evaluation  for  Concept 
Demonstration/  Validation  (D/V)  and  Full 
Scale  Development  (FSD). 

Designing  for  PJl  may  increase  the  initial 
acquisition  costs  of  the  system.  The  costs  of 
designing  in  the  flexibility  for  PJI  should  be 
documented  and  analyzed  in  terms  of  net 
utility  for  meeting  long  term  requirements. 
P-*I  requires  a  highly  disciplined  CM  system. 
Strict  configuration  control  must  be  applied 
to  such  areas  as  subsystem  boundaries, 
space,  weight,  power,  cooling  movement, 
centers  of  gravity,  electromagnetic  emissions, 
and  logistics  support  system.  Adequate 
post-deployment  communication  channels 
should  be  established  to  ensure  feedback  of 
modification  data  to  maintain  current 
configuration  status  accounting. 

In  the  initial  program  phases  (C/E  and 
DA').  systems  engineering  conducts 
functional  analyses  to  define  potential 
solutions  where  "add-on"  capability  is 
identified  as  necessary.  System  synthesis 
must  consider  the  additive  effects  of  these 
improvement  requirements;  e.g.,  will  the 
addition  of  extra  capabilities  in  a  fighter 
aircraft  cause  the  pilot  to  become 
overloaded  and  perform  less  efficiently  if  the 
initial  cockpit  design  uses  only  off-the-shelf 
components? 

General  P-*I  system  designs  could  include 
provisions  for: 

a.  Structure  -  Large  attachment  and 
support  strength  margins,  or  extra  volume 

b.  Power  -  Excess  power  generation, 


extra  power  bus  capability  to  accommodate 
additional  batteries  and  alternators,  or  spare 
wires  in  cables  (a  typical  retrofit  wire  on  an 
aircraft  passes  through  eight  bulkheads  and 
costs  $l,000/buikhead) 

c.  Cooling  -  Excess  pump  capacity,  area 
for  capability,  or  throat  area 

d.  Communications  -  Extra  radio 
frequency  power,  excess  band  width,  or  extra 
channels 

e.  Command  and  Control  -  Multiple  user 
data  bus,  large  computer  central  processing 
unit  margin,  or  excess  memory 

f.  Modular  Lay-Outs  -  Software  and 
hardware  units  partitioned  into  easily 
replaceable  pieces. 

A  good  example  to  illustrate  this  comes 
from  the  Air  Launched  Cruise  Missile 
(ALCM)  program.  It  was  determined  that 
the  range  of  the  ALCM  would  have  to  be 
increased;  therefore,  the  structural  strengths, 
supports,  and  internal  configuration  were 
designed  to  accommodate  the  potential 
increase  in  propellant  load.  When  increased 
range  requirements  became  achievable,  no 
significant  structural  design  changes  were 
required  to  accommodate  it.  Another 
ALCM  example  is  the  wing  (elevon)  design. 
The  elevons  are  designed  to  the  current 
ALCM  mission  requirement.  Knowing  that 
the  user  will  need  to  change  mission  profiles 
in  the  future  and  that  the  elevon  design  will 
have  to  change  as  a  function  of  altitude  and 
speed,  the  elevons  were  fastened  to  the 
ALCM  with  eight  accessible  bolts,  rather 
than  being  permanently  bonded  to  the 
structure.  This  simplifies  both  manufacturing 
assembly  and  field  modification 
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requirements. 

16.5  LESSONS  LEARNED  ON 

MODIFICATION  PROGRAMS 

The  following  are  some  of  the  more 
common  problems  associated  with 
modification  programs: 

Failure  to  Use  Below-Depot  Capability  - 

There  is  a  tendency  to  accomplish  all 
modifications  at  the  depot  level  when  many 
modifications  may  be  more  effectively  and 
economically  accomplished  at  lower  levels. 

Unreliable  Modification  Management  Data 

-  Records  of  those  units  already  modified, 
status  of  multiple  modifications  on  individual 
units,  and  other  related  data  are  incorrectly 
maintained  or  are  missing. 

Inadequate  Modification  Testing  -  The 
pressure  for  early  implementation  of  the 
modifications  sometimes  overrides  the  need 
to  adequately  test  new  modifications. 
Consequently,  some  modifications  get  to  the 
field  that  do  not  work  with  existing 
equipment  or  do  not  work  at  all.  In  either 
case,  the  situation  usually  requires  a 
modification  to  the  modification. 

Installation  Delays  -  By  the  time  some 
modification  kits  find  their  way  to  the  end 
item  in  the  field,  the  item  has  been  altered 
or  was  not  in  the  expected  configuration  to 
begin  with  (see  "Unreliable  Modification 
Management  Data,"  above).  Therefore, 
installation  of  the  modification  is  further 
delayed  while  a  change  is  obtained  so  that 
the  installation  may  be  accomplished. 

Inadequate  Kit  Accountability  and  Control 
Procedures  -  Bulk  purchases  of  modification 


kits  to  be  installed  over  an  extended  period 
of  time  expose  the  stored  kits  to 
cannibalization,  pilferage,  and  other  loss. 

Difficulties  in  Integrating  Changes  -  Poor 
communications  between  user,  developer, 
and  supporter  lead  to  mismatches  between 
modification  design,  end  item  design,  and 
user  needs.  This  makes  integration  one  of 
the  biggest  single  problems  with  modification 
programs. 

Inadequate  Support  Planning  -  There  is  a 
tendency  to  overlook  how  a  modified  system 
will  be  supported  and  how  multiple 
configurations  of  a  system  will  be  supported 
until  all  modifications  are  installed. 

16.6  DOCUMENTATION:  SYSTEM 
MODIFICATION  PLAN 

Preparation  of  a  modification  plan  is  a 
service  peculiar  process.  Essential  elements 
of  the  modification  plan  should  include: 

a.  Purpose  of  the  modification  -  impact 
on  system  effectiveness 

b.  Description  of  modification  -  power, 
weight,  volume,  data  interface 

c.  Identification  of  items  to  be  modified 

type,  location,  configuration  status, 

availability 

d.  Strategy  for  implementation 
schedule,  modification,  phasing 

e.  Organizational  responsibilities  -  lines 
of  communications,  CM,  data  generation 

f.  Logistics  support  -  packaging,  handling, 
transportation,  and  storage  (PHST),  training, 
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manuals,  and  supplies. 
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CHAPTER  17 


LIFE  CYCLE  COST 


17.1  INTRODUCTION 

Life  cycle  cost  (LCC)  is  the  total  cost  to 
the  government  of  acquisition,  ownership, 
and  disposal  of  a  system  over  its  entire  life. 
During  the  Concept  Exploration/  Definition 
(C/E)  phase,  the  LCC  effort  is  focused  on 
identifying  cost  drivers,  evaluating  relative 
LCC  differences  among  competing 
alternative  concepts,  and  developing  the 
estimate(s)  supporting  the  System  Concept 
Paper  (SCP)  supporting  Milestone  I.  During 
the  Concept  Demonstration/  Validation 
(D/V)  phase,  the  LCC  emphasis  is  on 
developing  a  LCC  estimate  for  each 
alternative.  During  Full  Scale  Development 
(FSD),  the  LCC  baseline  cost  estimate 
established  in  D/V  must  be  refined.  In  FSD, 
LCC  begins  its  transition  from  primarily  a 
design  element  to  a  control  element  for  the 
program.  All  decisions  should  still  be 
considered  in  light  of  their  effect  on  LCC, 
but,  at  this  point  in  development,  LCC  is 
more  of  a  control  tool  for  keeping  the 
program  on  track  by  highlighting  the  effect 
that  decisions  and  changes  will  have  on  total 
program  cost.  LCC  is  used  in  Operation  and 
Support  (O&S)  to  consider  the  impact  of 
modifications,  value  engineering  proposals, 
and  product  performance  agreements. 

DoDD  4245.3  [1]  is  the  principal  policy 
statement  on  cost.  The  directive  states  that 
cost  is  " a  parameter  equal  in  importance  to 
technical  and  supportability  requirements  and 


schedules.”  The  directive  calls  for  designing 
for  lower  LCC  by  establishing  cost  goals  and 
thresholds  based  on  credible  acquisition  and 
O&S  cost  parameters  that  are  consistent  with 
program  plans  and  budgets  and  that  achieve 
the  best  balance  among  cost,  schedule, 
performance,  reliability,  and  supportability 
characteristics.  Measurable  and  achievable 
design  to  cost  (DTC)  goals  and  thresholds 
must  be  established  at  Milestone  II. 

Historically,  a  low  initial  acquisition  cost 
for  hardware  has  not  assured  a  low  LCC.  In 
fact,  the  opposite  is  true.  The  bulk  of  LCC 
is  usually  in  O&S  costs,  as  shown  in  Figure 
17-1.  The  majority  of  system  costs  is 
typically  in  the  operations  and  support  area. 
Since  there  are  always  alternative  concepts 
and  design  for  system  support,  there  is  a 
need  to  assure  that  the  accepted  concept  and 
design  will  not  have  excessive  O&S  costs. 
This  sometimes  means  spending  more  on 
research  and  development  to  improve 
reliability  or  maintenance  access.  The  timing 
of  that  effort,  and  the  relative  impact  on 
LCC  however,  is  sensitive  to  the  evolution  of 
the  design. 

Figure  17-2  depicts  the  opportunity  for 
reducing  costly  design  concepts.  As 
illustrated  by  the  figure,  efforts  to  minimize 
life  cycle  costs  are  most  effective  in  the 
conceptual  and  early  design  stages  when 
alternatives  are  being  identified  and  selected. 
By  Milestone  I,  roughly  70  percent  of  a 


17-1 


Figure  17-1 

Nominal  Cost  Distribution  of  a  Typical  DoD  Program 
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system’s  LCC  is  "locked  in"  by  design 
decisions,  while  less  than  5  percent  of  its 
LCC  has  actually  been  expended. 

Although  the  military  budget  has 
remained  relatively  flat  (in  constant  dollars) 
during  the  past  30  years,  O&S  costs  have 
grown  during  that  period  at  a  rate  greater 
than  three  percent  per  year  (constant 
dollars).  During  this  period,  the  military 
budget  has  steadily  declined  as  a  percentage 
of  gross  national  product.  This  implies  that 
the  greatest  potential  opportunity  for  cost 
reduction  in  the  Department  of  Defense 
(DoD)  lies  in  controlling  the  high  yet 
invisible  cost  of  system  support. 

17.2  APPROACH 

Within  life  cycle  cost,  there  are  several 
well-defined  cost  components,  as  shown  in 
Figure  17-3.  One  used  often  between  the 
Program  Manager  (PM)  and  the  industry 


contractor  for  cost  control  is  "flyaway", 
"sailaway",  or  "rollaway"  cost.  This  is  the  cost 
of  procuring  the  basic  unit,  average  changes 
allowance,  cost  allocated  to  the  basic  unit, 
propulsion  equipment,  electronics,  armament, 
other  installed  government  furnished 
equipment  (GFE),  and  nonrecurring 
production.  Warranty  coverage  is  also 
included.  These  categories  are  defined  in 
DoDI  5000.33  [2]. 

Costs  for  support  equipment  required  to 
operate  in  the  field,  costs  for  data,  and  costs 
for  training,  when  added  to  flyaway  (sailaway 
or  rollaway)  cost,  make  up  "weapon  ystem" 
cost.  PROCUREMENT  cost  m  creases 
beyond  weapon  system  cost  to  include  the 
initial  spares.  Procurement  cost  added  to  the 
cost  of  research,  development,  test,  and 
evaluation  (RDT&E)  and  facility  construction 
costs,  makes  up  "program  acquisition"  cost, 
even  if  the  cost  for  facilities  is  not  in  the 
program  office  budget.  The  total  of  all  of 
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these  costs,  including  the  cost  of  operation, 
support,  and  eventual  disposal,  makes  up  life 
cycle  cost. 

Specific  terms  are  used  to  refer  to  various 
cost  elements.  Figure  17-4  displays  these 
cost  terms  by  program  phase.  Prior  to 
Milestone  II,  the  program  office  and 
contractor  establish  a  cost  objective  for  the 
program  in  terms  of  achievable  design  to  cost 
values.  During  FSD,  the  objective  becomes 
a  firm  goal  and  not-to-exceed  threshold  that 
the  program  office  accepts  in  the  Decision 
Coordinating  Paper  (DCP).  The  goal  and 
threshold  are  translated  into  a  target  for  the 
contractor  with  award  of  the  Milestone  III 
production  contract. 


173  ESTABLISHING  COST  GOALS 

A  design  to  cost  program  is  an  important 
subset  of  the  LCC  concept.  It  requires  a 
system  to  be  designed  and  built  within 
specific  cost  goals,  both  for  average  unit 
flyaway  cost  and  O&S  cost  parameters.  A 
DTC  program  seeks  to  balance  unit 
production  costs  against  ownership  costs 
during  development  while  decisions  can  still 
be  implemented.  The  Joint  Design  to  Cost 
Guide  [3]  states  that  DTC/LCC  should  be 
applied  early  in  the  development  cycle,  when 
the  design  may  be  heavily  influenced  through 
requirements  generation.  Just  prior  to  FSD, 
DTC/LCC  provides  information  upon  which 
to  base  decisions  for  alternative  concepts  and 
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Cost  Terms  by  Program  Phase 
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designs.  A  flyaway  (sailaway  or  rollaway) 
cost  goal  is  established  by  the  program  office 
based  on  such  factors  as  budgetary 
constraints  established  by  Congress,  service 
budgetary  constraints,  independent  cost 
estimates,  economic  forecasts,  prior  concept 
exploration  studies,  etc.  A  certain  portion  of 
flyaway  cost  is  allocated  to  the  contractor  as 
the  goal  for  unit  production  cost.  The 
program  office  will  retain  the  remainder  of 
the  flyaway  cost  to  cover  internal  DoD 
investment  costs  and  engineering  change 
allowances. 

Per  Reference  [3],  application  of  cost 
goals  in  FSD  is  mandatory,  since  FSD  is  the 
last  opportunity  to  significantly  influence  the 
design  before  production.  It  is  essential  that 
the  selected  cost  goals  can  be  related  directly 
to  the  LCC  estimates  that  support  the  DCP 
at  the  Milestone  II  review  or  budget 
submittals.  The  average  unit  flyaway  cost 
objective  becomes  a  firm  DTC  goal  after 
Milestone  II.  Firm  goals  and  thresholds  for 
O&S  parameters  must  also  be  established  at 
Milestone  II.  Selected  O&S  parameters  (e.g., 
reliability  and  maintainability)  should 
represent  factors  that  significantly  affect  O&S 
costs,  are  design  controllable,  and  can  be 
measured  during  test  and  evaluation. 

DTC  goals  should  be  established  from  a 
knowledge  of  cost  improvement  potential  for 
the  system,  together  with  projected  budgetary 
limitations.  The  key  to  achieving  DTC  goals 
is  flexibility  in  allowing  the  designer  freedom 
of  choice  and  decision  to  arrive  at  a 
configuration  that  satisfies  mission  objectives. 
This  can  be  achieved  through  contracts  and 
specifications  that: 

a.  Specify  the  performance  needed,  not  the 


way  to  obtain  the  result 

b.  Specify  a  total  time  to  operational 
capability,  not  detailed  interim  milestones 

c.  Schedule  a  program  permiting  several 
iterations,  not  on  a  100  percent  success  basis 
for  the  first  attempt. 

DTC  goals  should  be  achievable.  If  the 
goal  is  too  easily  achieved,  there  is  no 
motivation  toward  cost  reduction  through 
critical  examination  of  requirements, 
concepts,  and  designs.  This  may  result  in 
acquiring  incremental  performance  or  design 
features  that  are  not  cost-effective. 
Conversely,  if  the  goal  is  too  difficult, 
motivation  is  destroyed  because  no  amount 
of  ingenuity  on  the  part  of  the  contractors  to 
use  advanced  technology  or  simplified  designs 
could  be  expected  to  achieve  the  goal. 

17.4  COST  ESTIMATING  PROCEDURES 

LCC  may  be  estimated  using  any  of  a 
number  of  estimating  techniques.  The  cost 
estimate  evolves  and  is  refined  as  the 
program  matures.  This  is  illustrated  in 
Figure  17-5.  The  source  and  type  of  the  cost 
estimate  are  usually  dependent  on  the 
certainty  of  the  cost  elements.  The  following 
estimating  techniques  are  most  commonly 
used: 

a.  Parametric  analysis 

b.  Analogy 

c.  Bottom-up  or  engineering 

d.  Other  (best  guess,  Bayesian 
Statistics/Delphi  Process). 
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Figure  17-5 

Cost  Estimating  Methods  and  Most  Prevalent 
Use  by  Acquisition  Phase 
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On  large  programs,  all  of  these 
approaches  may  be  used  to  provide  an 
estimate  of  costs.  However,  bottom  up 
estimating  is  labor  intensive  and  time 
con'’~  ling.  Parametric  analysis  can  be  done 
vt  uickly  (in  a  matter  of  days)  and 
con  ues  closely  with  estimates  generated 
using  .e  bottom  up  method  (usually  within 
two  to  three  percent).  The  analogy  method 
is  gener^’y  used  to  calibrate  results  of  the 
paranu  analysis.  Noie  that  the  "best 
guess"  '  lot  intended  to  be  a  guess  for  the 
total  system  cost.  Frequently,  estimating  the 
cost  for  a  component  where  no  precedent 
data  are  available  requires  an  educated 
guess. 

17.4.1  Parametric  Analysis 

A  broad  existing  data  base  gathered  from 


hardware  and  software  developed  for  similar 
purposes  can  be  manipulated  by  a  computer 
to  establish  a  price,  using  only  a  minimum 
number  of  characteristics  (such  as  weight  and 
size).  Several  computer  based  parametric 
models  are  available  to  perform  cost  analysis, 
as  shown  in  Figure  17-6.  However,  the  RCA 
PRICE  (Programmed  Review  of  Information 
for  Costing  and  Evaluation)  model  has  by  far 
the  widest  use  and  is  now  employed  by  the 
Air  Force,  Army,  Navy,  NASA,  aerospace 
companies,  many  commercial  organizations, 
and  several  foreign  governments,  to  provide 
cost  estimates. 

17.4.2  Analogy 

Where  the  new  item  has  functions  and 
physical  and  performance  characteristics 
similar  to  an  existing  item,  current  costs  on 
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Figure  17-6 
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the  similar  item  can  be  gathered  and 
modified  appropriately  to  account  for  the 
difference  in  configuration. 

17.43  Bottom  Up 

Each  organization  and  group  involved  in 
the  development  and  production  of  an  item 
estimates  its  costs.  The  results  are  totaled 
and  combined  with  overhead,  general  and 
administrative  expense,  and  contract  fee  to 
arrive  at  an  estimate  for  the  item.  Based  on 
a  preliminary  equipment  list,  test  concepts, 
and  a  Manufacturing  Plan,  the  cost  of 
hardware  items  is  established  for 
development  and  production.  Lines  of  code 
are  estimated  based  on  the  identification  and 


definition  of  software  modules.  Software 
costs  can  then  be  established  for 
development  and  testing.  Operational  costs 
and  costs  associated  with  spares,  support 
equipment,  training,  data,  operating 
personnel  requirements,  and  supplies  and 
material  for  the  prescribed  operational  period 
are  also  defined.  The  aggregate  of  these 
costs  is  the  cost  to  develop  and  field  the 
entire  system. 

17.5  LCC  ANALYSIS 

LCC  analysis  is  the  structured  study  of 
LCC  estimates  and  elements  to  identify  life 
cycle  cost  drivers,  total  cost  to  the 
government,  cost  risk  items,  and  cost 
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effective  changes.  It  is  a  systems  engineering 
tool  with  application  to  all  elements  of  the 
system. 

Computer  modeling  is  often  used  to 
identify  and  analyze  life  cycle  cost  drivers. 
Cost  drivers  are  parameters  that  control  O&S 
costs  such  as  reliability,  maintainability,  parts, 
and  support  equipment,  to  name  a  few.  Cost 
drivers  are  areas  where  resources  can  best  be 
applied  to  achieve  the  greatest  benefit  in 
reduced  costs.  For  example,  a  reduction  in 
manpower  tequirements  would  have  a 
significant  effect  on  a  system’s  LCC.  An 
effective  LCC  analysis  will  identify  areas 
where  contract  incentives  may  be  applied  to 
earn  the  greatest  pay-off. 

Modeling  for  LCC  is  also  useful  in  cost 
benefit  and  cost  effectiveness  studies, 
long-range  planning  and  budgeting, 
comparison  of  competing  systems,  decisions 
about  replacement  of  aging  equipment, 
control  of  an  ongoing  program,  and  selection 
among  competing  contractors. 

Many  computer  models  are  available,  but 
no  single  model  suits  every  application. 
Reference  [4]  offers  models,  calculations,  and 
formulas  (cost  estimating  relationships) 
appropriate  to  each  phase  of  the  acquisition 
cycle.  Reference  15]  evaluates  several  Air 
Force  computer  models  for  aviation  systems. 
The  Defense  Systems  Management  College 
(DSMC)  has  three  new  models  that  can  be 
run  on  a  personal  computer:  1)  a  LCC 
model,  CASA  (with  a  risk  model);  2)  a  quick 
reaction  model,  DPESO;  and  3)  a  schedule 
and  cost  risk  model,  VERT.  Each  of  the 
models  provides  a  means  for  examining  cost 
impact. 


17.6  APPLICATION  OF  TRADE-OFF 
ANALYSES  TO  LCC/DTC 

LCC  trade-off  analyses  are  employed  to 
obtain  an  optimum  balance  between  cost  and 
effectiveness.  The  trade-off  analysis  method 
is  discussed  in  Chapter  8  of  this  guide. 
Potential  input  variables  for  LCC  trade-off 
analyses  are  shown  in  Figure  17-7.  As  an 
example,  microprocessors  could  be 
incorporated  into  the  design  to  automate 
functions  and  pn  dde  for  built-in-test  (BIT). 
This  would  increase  development  cost  (due 
to  software  develo  -nent)  but  could  reduce 
weight,  size,  and  maintenance  costs  with  a 
potential  overall  cost  reduction.  In 
considering  total  LCC,  reliability  and 
maintainability  have  a  major  impact  since  the 
system  and  subordinate  level  unreliability 
reflect  directly  on  required  logistic  support  in 
terms  of  skills  and  levels  of  maintenance 
personnel  required,  spares  stackage, 
personnel  training,  and  their  effect  on 
operation  and  maintenance  (O&M)  cost. 

The  DTC  trade-off  analyses  take  into 
consideration  the  variables  of  reliability  and 
maintenance  at  some  sparing  level,  and  the 
DTC  aspects  of  each  trade-off  candidate.  At 
one  extreme,  a  highly  reliable  system  may 
represent  a  high  order,  front  end  investment. 
This  investment  will  be  reflected  through  all 
subordinate  system  elements  of  hardware  and 
software.  At  the  other  extreme,  candidates 
of  nominal  reliability  represent  a  far  lower 
investment,  but  may  be  expected  to  incur 
higher  logistics  and  maintenance  costs  in 
end-use  service.  The  DTC  objective  is  set  at 
the  optimum  point  between  these  extremes. 

As  part  of  the  maintenance  trade-off 
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Figure  17-7 

Trade-Off  and  input  Variables 
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•  Power  Reque  ament* 

analyses,  the  following  variables  may  be 
examined  to  establish  cost  drivers: 

a.  Number  of  equipment  level  maintenance 
and/or  supply  locations 

b.  Number  of  organization,  intermediate, 
and  depot  level  maintenance  locations 

c.  Number  and  level  of  skills  required  for 
support 

d.  Duration  of  the  support  period 


e.  Equipment  operating  time 

f.  Number  of  line  replaceable  units  (LRUs), 
modules,  and  parts 

g.  LRU  mean  time  between  failure 

h.  LRU  and  module  mean  time  to  repair 

i.  Amount  and  quality  of  built-in-test 
equipment  (BITE) 

j.  Cost  of  contractor  repair 
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Sensitivity  studies  are  conducted  to 
identify  areas  where  resources  can  best  be 
applied  to  achieve  the  maximum  cost  benefit. 
These  studies  examine  performance 
parameters  to  determine  where  small  changes 
in  the  parameters  would  produce  significant 
changes  in  development  and  operational 
costs.  In  the  example  shown  in  Figure  17-8, 
a  relatively  small  change  in  maintenance 
manhours  per  flight  hour  or  mean  time 
between  overhaul  results  in  large  savings  in 
operational  costs. 

q.  Cost  of  entering  and  maintaining  items  17.7  DOCUMENTATION 
in  the  supply  system 

It  is  the  PM’s  responsibility  to  ensure  that 

r.  Cost  of  programming  and  documenting  the  contractor  establishes  and  carries  out  a 

test  equipment  LCC  program.  An  assessment  of  how  well 

the  LCC  program  is  implemented  should  be 

s.  Dedicated  versus  non  dedicated  crews.  made  during  program  management  reviews 

Figure  17-8 

Cost  Sensitivity  Study  Results  for  an  Aircraft  Program 


k.  Test  equipment  costs 

l.  Spares  and  test  equipment  storage  space 
requirements 

m.  Production  learning  curves 

n.  Safety  stock  coefficients 

o.  Resupply  times 

p.  Crew  size  and  labor  rates 


and  technical  reviews,  and  prior  to  key 
program  demonstration  milestones.  During 
program  management  reviews,  the  contractor 
should  be  masked  to  address:  1)  life  cycle  cost 
implementation  plans  and  the  status  of  key 
activities;  2)  cost  drivers  and  actions  taken  or 
actions  planned  to  reduce  or  control  costs;  3) 
the  status  of  cost-related  design  goals;  and  4) 
trade-off  analyses  planned,  ongoing,  or 
recently  completed. 

Supporting  documentation  for  the 
contractor’s  program  is  provided  in  the  Life 
Cycle  Cost  Plan  (LCCP).  The  contractor’s 
LCC  estimate,  associated  Trade  Study 
Reports  (TSRs),  and  cost  related  design  goal 
status  data  should  also  be  reviewed  by  the 
program  office. 

17.7.1  Life  Cycle  Cost  Plan 

The  LCCP,  as  required  by 
MIL-STD-499A  [6],  is  developed  by  the 
contractor  to  describe  the  approach  for 
integrating  LCC  into  the  management  and 
design  effort.  The  plan  should  address  the 
following  issues  (as  recommended  by 
Reference  (4]): 

a.  On  what  ground  rules  and  assumptions 
should  the  LCC  analysis  be  based? 

b.  Which  estimating  procedures  will  be 
used? 

c.  What  kind  of  product  is  the  LCC  analysis 
to  produce,  for  whom,  in  what  format,  and 
for  what  purpose? 

d.  How  will  buyer  and  seller  audit  and 
control  the  LCC  process? 


e.  How  will  the  LCC  effort  be  organized 
and  financed? 

Recommendations  for  content  of  the 
LCCP,  as  defined  by  data  item  description 
DI-F-30203  (7],  include  the  following: 

a.  Statement  of  the  contractor’s  LCC 
management  objectives  and  a  description  of 
supporting  tasks,  milestones,  and 
responsibilities 

b.  Program  structure,  policies,  procedures, 
and  functional  relationships  for  maintaining 
LCC  visibility  and  control 

c.  Method(s)  for  determining  and 
identifying  LCC  drivers  and  issues  subject  to 
trade-off  analyses 

d.  Preliminary  list  of  the  10  most  influential 
contract  requirements  that  affect  the  LCC 
of  the  system  (c.g.,  performance,  schedules, 
standards,  specifications) 

e.  Description  of  planned  analysis  methods 
and  DTC/LCC  modeling  techniques  to  be 
used  in  LCC  analysis 

f.  Management  approach  for  integrating 
subcontractors’  effort  into  LCC  management 
efforts 

g.  Recommended  LCC/DTC  goals  and 
planned  allocation  procedures 

h.  Planned  feedback  mechanism  for 
tracking  and  supporting  cost  related  design 
goals  and  status,  including  proposed  analysis, 
test,  and  evaluation  efforts  to  be  used  as 
progress  checks. 
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17.7.2  Life  Cycle  Cost  Estimate 

The  contractor  should  provide  a  LCC 
estimate  that  will  serve  as  a  cost  baseline  for 
the  program.  The  documentation  of  this 
estimate  should  include: 

a.  Purpose  and  scope 

b.  System  and  program  description 
summary 

c.  Program  schedule  summary 

d.  Ground  rules  and  assumptions 

e.  Summary  estimates  for  RDT&E, 
production,  and  O&S  costs 

f.  Rank  ordered  list  of  systems/ 
components/  software,  which  account  for  not 
less  than  80  percent  of  the  total  estimated 
system  LCC 

g.  RDT&E  estimate  by  work  breakdown 
structure  (WBS)  element  and  function 

h.  Production  estimate  by  WBS  element 
and  function 

i.  O&S  cost  by  WBS  element  and  function 

j.  Time-phased  program  costs 

k.  Funding  spreads 

l.  Inflation  and  discounting  methodology 
and  indices 

m.  LCC  estimate  tracking 

n.  sensitivity  analyses 


o.  Risk  and  uncertainty  analysis  of  each 
WBS  level. 

Adequate  documentation  requires  three 
basic  elements:  1)  the  data  and  sources  of 
data  on  which  the  estimate  is  based,  2)  the 
estimating  methods  applied  to  that  data,  and 
3)  the  results  of  the  analysis.  The  main 
thrust  of  any  documentation  package  is 
methodology.  Parametric  equations,  learning 
curves,  cost  performance  analyses,  and  factor 
derivations  or  build-up  techniques  for  each 
part  of  the  estimate  are  described  to  provide 
continuity  and  consistency  and  to  facilitate 
tracking  for  future  estimates.  An  estimate 
track  consists  of  a  comparison  to  prior 

estimates  and  an  analvsis  of  reasons  for 

* 

differences.  The  explanation  of  differences 
should  be  quantitatively  expressed,  if  possible. 

17.73  Engineering  Trade  Study  Report 

An  engineering  Trade  Study  Report 
documents  the  trade-off  analyses  conducted 
to  achieve  cost  goals.  It  shouid  describe  the 
analysis  methods  and  their  adequacy,  identify 
data  sources  and  degree  of  confidence  in  the 
accuracy  of  the  data,  and  define  alternatives 
and  the  rationale  for  selection.  Cost  drivers 
should  be  defined  and  areas  for  future 
trade-off  analyses  identified. 

17.7.4  Cost-Related  Design  Goal  Status  Data 

Design  goal  status  data  arc  produced 
during  the  FSD  phase  and  used  to  measure 
performance  toward  achieving  the  cost 
related  design  goals,  and  to  provide 
projections  and  analyses  necessary  to  develop 
timely  management  decisions  concerning 
trade-off  and  design  changes.  For  the  unit 
production  cost  goal,  status  information  is 
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normally  provided  at  the  summary  WBS  level 
and  for  each  specified  WBS  element  in  terms 
of  hours  and  dollars  for  each  functional  cost 
(e.g.,  engineering,  tooling,  manufacturing, 
quality  control,  and  purchased  equipment)  for 
recurring  and  non-recurring  cost.  For 
non-dollar,  cost  related  design  goals  such  as 
crew  size,  maintenance  manpower,  and 
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CHAPTER  18 


MANUFACTURING  AND  PRODUCIBIUTY 


18.1  INTRODUCTION 

One  conclusion  that  may  be  drawn  from 
DoDD  4245.7  [1]  is  that  production 
management  is  a  system  management  job. 
It  is  no  accident  that  published  descriptions 
of  the  systems  engineering  process  since  the 
early  1960s  have  included  the  production 
function  as  an  integral  part  of  the  system  life 
cycle.  Failure  to  consider  production 
requirements  early  in  the  acquisition  cycle 
has  resulted  in  more  than  acceptable  risk  in 
the  transition  from  development  to 
production.  As  pointed  out  in  the 
Department  of  Defense  (DoD)  study  [2], 
"The  acquisition  process  (should  be)  recognized 
for  what  it  is  ...  an  industrial  process  concerned 
with  the  design,  test,  and  production  of  low  risk 
products ".  Product  design  and  development 
cannot  be  logically  separated  from  the 
production  process.  The  approach  taken  to 
design  most  often  radically  constrains  the 
producibility  of  a  product.  It  is  for  this 
reason  that  producibility  attributes  should  be 
an  integral  part  of  all  configuration  item 
trade  studies. 

Once  production  is  viewed  as  simply 
another  life  cycle  function,  not  unlike 
companion  operation  and  support  functions, 
then  systems  engineering  tools  can  be  used 
to  manage  the  production  process.  The 
timely  application  of  systems  engineering 
tools  and  the  systems  engineering  process 


can  contribute  to  assisting  industry  in 
achieving  higher  levels  of  productivity. 

The  capability  to  produce  a  hardware 
item  that  satisfies  mission  objectives  is  as 
essential  to  the  systems  engineering  process 
as  other  functions  such  as  operations  or 
support.  The  application  of  the  systems 
engineering  process  to  production  functions 
will  identify  the  products  required  to 
transform  design  into  a  capability  for 
efficient  and  economical  production  of 
equipment  and  facility  elements  of  the 
system.  Systems  engineering  also  ensures 
that  production  -capabilities  are  constantly 
used  as  design  selection  criteria.  Typical 
production  functions  that  will  be  analyzed 
during  a  fully  integrated  design  effort  include 
such  actions  as  material  ordering,  material 
handling,  fabrication,  processing,  quality 
assurance,  process  control,  assembly, 
inspection,  test,  preservation,  packaging, 
storage,  shipping,  and  disposition  of  scrap, 
salvage,  and  waste  materials.  All  of  this 
analysis  is  best  done  under  the  Total  Quality 
Management  (TQM)  umbrella. 

The  systems  engineering  process  applied 
to  production  functions  supports  producibility 
analyses,  production  engineering  inputs  to 
trade  studies,  life  cycle  cost  analyses,  and 
consideration  of  the  materials,  tools,  test 
equipment,  facilities,  personnel,  software, 
and  procedures  that  support  manufacturing 
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Figure  18-1 

Manufacturing  Activities  in  the  System  Acquisition  Process 


Concept  Exploration/  Definition  Phase 
•Evaluate  Production  Feasibility 
•Assess  Production  Risk 

•Identify  Manufacturing  Technology  Needs 
•Identify  Manufacturing  Cost 
•Develop  Manufacturing  Strategy 
•Identify  Deficiencies  in  U.S.  Industrial  Base 
•Determine  Availability  of  Critical  Materials 
•Develop  Contract  Requirements  for  D/V  Phase 
•Begin  Value  Analysis  Studies 
Concept  Demonstration/  Validation  Phase 

•Assess  Producibility  of  Competitive  Designs 
•Accomplish  Production  Risk  Resolution 
•Reassess  Production  Transition  Risk 
•Evaluate  Producibility  Criteria 
•Plan  for  Achieving  Producibility 
•Assess  Production  Feasibility 

•Complete  Manufacturing  Technology  Developments 
•Plan  for  Use  of  Competition  in  Production 
•Develop  Initial  Manufacturing  Plan 
•Evaluate  Long  Lead  Procurement  Requirements 
•Develop  Initial  Manufacturing  Cost  Estimate 
•Develop  Production  Readiness  Review  Plan 
•Develop  Contract  Requirements  for  FSD  Phase 
•Continue  Valua  Analysis/  Engineering  Efforts 
Full  Scale  Development  Phase 

•Evaluate  Producibility  of  Design 
•Revise  Production  Risk  Evaluations 
•Define  Required  Manufacturing  Resources 
.Develop  Detailed  Production  Design 

•Define  and  Proof  Manufacturing  Processes  and  Equipment 

•Accomplish  Producibility  Engineering 

•Accomplish  Production  Planning 

•Integrate  Spares  Production 

•Develop  Production  Work  Breakdown  Structure 

•Develop  Manufacturing  Cost  Estimates 

•Complete  Manufacturing  Plan 

•  Plan  for  and  Accori'lish  System  Transition 

•Accomplish  Product  ion  Readiness  Reviews 

•Develop  Contract  Requirements  for  Production  Phase 

•Complete  Initial  Production  Facilities 

•Solidify  Production  Baseline/  Limit  Engineering  Changes 
Production  and  Deployment  Phase 

•Execute  Manufacturing  Program 
•Maintain  Production  Surveillance 

•Implement  Product  Improvements  as  planned  for  in  FSD 
. Provide  and  Support  Government-Furnished  Property 
.Accomplish  Further  Value  Engineering  (e.g.,  on  P3I,  processes) 
•Accomplish  Second  Sourcing/ Component  Break-Out 
•Complete  Industrial  Preparedness  Planning 
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in  the  Concept  Demonstration/  Validation 
(D/V)  and  Production  phases.  For  example, 
critical  producibility  requirements  are 
identified  early  and  incorporated  into  the 
program  risk  analysis.  Where  the  production 
functional  analysis  identifies  requirements 
creating  a  constraint  on  the  design,  they  are 
included  in  applicable  development  and 
product  specifications  and/or  engineering 
drawings.  Programs  involving  an 

evolutionary  upgrade  as  well  as  higher  risk 
efforts  should  infuse  the  producibility  process 
as  early  as  concept  exploration. 

This  chapter  describes  the  tools  and 
procedures  with  which  production 
engineering,  as  an  integral  part  of  the 
systems  engineering  "team,"  interacts  with 
designers  to  ensure  that  the  resulting  design 
represents  the  most  producible  design  with 
acceptable  cost  and  schedule  risk. 
Production  engineering  is  defined  in  the 
Defense  Systems  Management  College 
(DSMC)  Program  Manager’s  (PM) 
Notebook  as  the  application  of  design  and 
analysis  techniques  to  produce  a  specified 
product,  including: 

a.  The  functions  of  planning,  specifying, 
and  coordinating  the  application  of  required 
resources 

b.  Performing  analyses  of  producibility 
and  production  operations,  processes,  and 
systems 

c.  Applying  new  as  well  as  existing 
manufacturing  methods,  tools,  and 
equipment 

d.  Controlling  the  introduction  of 


engineering  changes  so  that  they  are  almost 
non-existant  by  Milestone  III  startup 

e.  Employing  meaningful  cost  control 
techniques. 

18.2  APPROACH 

Manufacturing  activities  begin  in  the 
Concept  Exploration/  Definition  (C/E)  phase 
when  a  system  concept  has  been  defined. 
Initial  activities  are  concerned  with 
production  feasibility,  costs,  and  risks.  Prior 
to  Milestone  I,  DoDI  5000.2  [3]  requires 
that  an  industrial  resource  analysis  (IRA)  be 
conducted  to  determine  the  availability  of 
production  resources  required  to  support  a 
major  system  production  program.  These 
resources  include  capital,  material,  and 
manpower  required  to  accelerate  and 
maintain  full  production  ratios  and  respond 
to  surge  and  mobilization  requirements.  The 
IRA  would  include  results  of  feasibility 
studies,  producibility  analyses,  and 
manufacturing  program  and  producibility 
assessments.  As  development  proceeds, 
trade  studies  and  preaward  surveys  are 
conducted  to  establish  the  most  cost-effective 
methods  for  manufacturing  items,  and 
detailed  plans  are  developed  for  the 
Production  Phase.  Prior  to  Production, 
extensive  controls  are  implemented  at  both 
prime  contractor  and  subcontractor  facilities 
to  ensure  that  the  product  will  meet 
specifications.  Producibility  analyses  may 
generate  the  need  for  a  requirements  scrub 
effort  by  the  PM  and  user  during  the  C/E  or 
D/V  phases. 

MIL-STD-499A  [4]  describes  a  basic 
systems  engineering  process  which  details 
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Figure  18-2 

Production/Systems  Engineering 
Life  Cycle  Relationships 


many  activities  that  support  production 
engineering,  producibility  engineering  and 
planning,  production  risk  management,  and 
general  integration  of  production 
requirements  within  the  engineering 
management  process.  Requirements  and 
options  for  production  rates  and  quantities 
are  elemental  drivers  of  the  systems 
engineering  controlled  design  process. 

The  requirements  for  contractor 
production  management  are  given  in  DoDD 
4245.6  [5].  It  defines  the  need  to  establish: 

a.  Industrial  resource  analyses 

b.  Production  Readiness  Reviews  (PRR) 

c.  Production  risk  analysis 

d.  Manufacturing  strategy 

e.  Comprehensive  producibility 
engineering  and  planning  program 

f.  Effective  integration  with  the  quality 
program 

g.  Independent  assessment  of  production 
readiness 

h.  Planning  for  post-production  activity 

i.  Incorporation  of  a  variety  of  cost 
avoidance  and/or  reduction  techniques 

j.  Emphasis  on  life  cycle  cost. 

The  risk  management  templates  of  DoD 
4245.7-M  [2]  are  designed  to  permit  ease  in 
the  management  of  the  systems  engineering 


process  so  that  it  ensures  low-risk 
production.  The  templates  are  to  be  used 
as  a  guide  and  tailored  to  the  individual 
production  programs. 

During  the  early  program  phases,  the 
contractor’s  production  engineering 
personnel  must  be  integrated  into  the 
systems  engineering  organization  to  ensure 
that  producibility  requirements  are 
incorporated  into  the  basic  systems 
engineering  documentation,  specifications, 
and  plans.  The  production  engineers  review 
conceptual  designs  together  with  other 
engineering  specialists,  conduct 
manufacturing  trade  studies  to  establish  the 
most  producible  design,  identify  required 
production  resources,  and  prepare  the 
Production  Plan. 

During  the  Full  Scale  Development 
(FSD)  phase,  a 'manufacturing  organization 
is  established  if  not  already  in  place  for 
other  programs.  The  elements  of  production 
engineering,  product  assurance,  planning, 
facilities,  and  production  equipment  are 
often  integrated  into  the  organization. 
Product  assurance  provides  process  control 
of  manufacturing  and  subcontractor 
operations  under  the  Total  Quality 
Management  umbrella.  Prototype  and 
qualification  articles  are  produced  and  tested 
to  demonstrate  that  the  system  meets  its 
specification  requirements.  Prior  to  the  start 
of  full  production,  Production  Readiness 
Reviews  are  conducted  to  assure  that  all 
necessary  resources  and  controls  are 
established  and  all  action  items  are  closed 
prior  to  MS  III.  Depending  on  tiiv.  and 
complexity  of  the  system,  a  PRR  may  be 
held  as  a  single  review  or  as  a  series  of 
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Figure  18-3 

Basic  Production  Systems  Engineering  Process 


reviews. 


183  PRODUCTION  ENGINEERING 
ANALYSIS 

Production  engineering  analysis  begins  in 
the  C/E  phase,  as  stated  in  MIL-STD-499A 

[4] ,  MDL-STD-1528  [6],  and  DoDD  4245.6 

[5] .  This  analysis  requires  rate  and  quantity 
inputs  that  may  themselves  be  the  products 
of  major  trade-off  analysis  efforts.  Typically, 
the  production  engineering  analysis  is 
performed  as  a  team  effort  to: 

a.  Establish  estimates  of  the  production 
capability  required 

b.  Assess  previous  production  and  user 
experience  and  problems  encountered  on 
similar  programs  in  conjunction  with  cost/ 
schedule  control  system  (C/SCS)  reporting 


c.  Identify,  develop,  and  document  new 
technology  or  special  processes 

d.  Assess  production  feasibility  and 
identify  risk  areas  including  use  of  the 
GIDEP  Reports. 

e.  Develop  production  costs  and 
schedules 

f.  Define  production  risk  mitigation 
approach  and  associated  milestones 

g.  Define  tooling  requirements  with 
worker  involvement 

h.  Define  a  production  test  plan 

i.  Establish  inspection  requirements  with 
worker  involvement 

j.  Establish  personnel  skills  and  training 
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requirements 

k.  Evaluate  existing  facilities  and  equipment 
to  establish  any  modifications  or  new 
resources  for  manufacturing 

l.  Develop  a  manufacturing  assembly 
sequence  chart 

m.  Define  and  implement  producibility 
criteria  into  the  technical  data  package 

n.  Identify  trade  areas  to  reduce  risk  or 
cost. 

The  objective  of  the  production 
engineering  analysis,  considered  as  an 
integral  part  of  the  systems  engineering 
process,  should  be  to  permit  the  production 
of  a  quality  system  (which  meets  the  user’s 
needs)  on-time,  at  the  lowest  possible  cost. 
The  basic  systems  engineering  process  has 
the  tools  and  documentation  to  effect 
definition  and  development  of  system 
elements  (equipment,  software,  personnel, 
procedures)  related  to  production  functions 
and  prod  action  requirements. 

Tvui.ng  the  arty  program  phases, 
pio  '  action  engineers  work  with  systems 
engine  ers  to  define  the  impact  on  existing 
resources  and  provide  data  on  manufacturing 
alternatives  to  proposed  designs  using  the 
basic  systems  engineering  process. 

18.4  PRODUCIBILITY  TRADE  STUDIES 

Manufacturing  trade  -  any  areas  include 
engineering  design,  reliability,  maintainability, 
program  schedules,  life  cycle  cost, 
effectiveness,  producibility,  supportability, 


and  other  factors  affecting  overall  program 
objectives.  Trade  studies  are  conducted  to 
evaluate  the  most  cost-effective 
manufacturing  process  to  be  employed  within 
program  constraints.  The  trade  study 
process  involves  the  identification  of 
alternate  candidates,  definition  of  evaluation 
criteria,  weighting  and  scoring  of  the 
candidates,  and  examination  of  adverse 
consequences.  Through  analysis  and 
gathering  of  data,  the  characteristics  of  each 
alternative  will  be  established.  The 
candidates  will  then  be  scored  and  the 
results  summarized  in  the  trade  table. 

18.5  MANUFACTURING  PLANNING 

The  results  of  the  production  engineering 
analysis  are  documented  in  the  Production 
Plan,  which  defines  manufacturing  concepts 
and  methods.  The  Production  Plan 
MIL-STD-1528  [6]  provides  sufficient 
information  to  supporting  organizations  to 
assure  a  timely,  coordinated  approach  to  the 
production  process.  The  plan  is  developed 
in  preliminary  form  during  the  C/E  and  D/V 
phases  and  is  part  of  a  CDRL.  The  final 
plan  is  completed  prior  to  the  PRR  effort. 
An  outline  of  the  plan  is  provided  in  Section 
18.7.  During  the  FSD  phase,  as  the  detailed 
design  is  completed  and  prototype  hardware 
is  developed,  production  engineering 
supports  planning  by  continuing  to  refine  its 
analyses  to  more  detailed  levels  and  by 
developing  requirements  for  items  not  visible 
in  earlier  phases,  such  as  shop  aids  or 
templates  that  could  optimize  production 
and  assembly.  After  the  baseline  design  is 
established,  engineering  change  proposals 
(ECPs)  are  evaluated  by  production 
engineering  as  part  of  the  configuration 
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Figure  18-4 

Reflector  Fabrication  Trade  Table 
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management  process  to  provide 
manufacturing  inputs  on  cost  and  schedule 
impacts. 

The  PRRs  are  conducted  during  the  FSD 
phase  to  establish  that  the  system  is  ready 
for  efficient  and  economical  quantity 
production,  that  adequate  test  planning  has 
been  accomplished,  and  that  problems 
encountered  have  been  resolved.  Each  PRR 
is  conducted  in  accordance  with  DoDI 
5000.38  [7]  and  makes  the  following 
evaluations: 

a.  Production  Design 

1.  Producible  at  low  risk 

2.  Design  change  rate  stabilized  at  low 
level 

3.  Design  validated:  performance, 

reliability,  maintainability,  availability 

4.  Design  deficiencies  identified 

5.  Design  standardized  and  stabilized 

b.  Resources 

1.  Plant  capacity  adequate 

2.  Skilled  personnel  available 

3.  Training  programs  available 

c.  Production  Engineering  and  Planning 

1.  Production  Plan  developed 

2.  Schedule  compatible  with  configuration 


item  delivery  requirements 

3.  Manufacturing  methods  and  processes 
integrated  with  facilities,  equipment,  tooling, 
and  lay-out 

4.  Production  planning  complete 

5.  Value  engineering  applied  and  large 
dollar  initiatives  incorporated  into  the  TDP 

6.  Configuration  management  adequate 
and  a  baseline  is  established 

7.  Management  information  system 
adequate 

d.  Materials  and  Purchased  Parts 

1.  Bill  of  materials  complete 

2.  Make-or-bay  decisions  complete 

3.  Long  lead  items  identified  and  ordered 

4.  Government-furnished  equipment 
identified  and  its  Total  Quality  control  and 
disposition  is  agreed  to  by  both  parties. 

5.  Inventory  control  system  adequate 

e.  Total  Quality  Management 

1.  Process  control  tools  are  in  place  and 
understood 

2.  Statement  of  work  (SOW)  satisfied 
which  includes  continuous  improvement 
practices 

3.  Acceptance  criteria  exists  with 
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emphasis  on  the  narrowing  of  variability  as 
a  goal 

4.  Subcontractor  control  including  flow 
down  of  TQM  and  its  tools. 

5.  Management  and  worker  involvement 

6.  On  going  training  at  all  levels 

7.  Problem  Prevention 

8  User  Involvement 

f.  Logistics 

1.  Operations  support,  test,  and 
diagnostic  equipment  available 

2.  Training  aids  and  simulators  available 

3.  Packaging,  handling,  storage,  and 
transportation  (PHST)  adequate 

4.  Spares,  Foreign  Military  Sales  (FMS) 
requirements,  and  life  of  type  buys 
incorporated 

g.  Software 

1.  Plans  complete 

2.  Rate-of-change  stable 

3.  Configuration  management  adequate 

4.  Security  adequate 

h.  Safety 

1.  Plans  complete  and  implemented 


18.6  PRODUCTION  FACILITIES 
IDENTIFICATION 

Although  producibility  studies  and  plans 
may  define  a  more  efficient  structure  for  the 
production  process,  implementation  of 
facility  solutions  is  often  constrained  by 
defense  contracting  procedures.  Defense 
contracting  procedures  can  inhibit  industry 
investments  to  modernize  because  profits 
may  be  reduced  when  costs  to  produce  go 
down.  Careful  contract  stucturing  can 
eliminate  this  problem.  The  contractor  if 
he’s  any  good  will  do  this  anyway. 

Factory  improvements  place  equal 
emphasis  on  all  system  elements.  This  often 
leads  to  interesting  conclusions.  For 
example,  a  key  feature  of  the  DoD  risk 
reduction  program  for  transition  from 
development  to  production  (outlined  in  DoD 
4245.7-M  (11)  is  the  identification  of  a 
general  need  for  productivity  centers  that 
provide  training  and  training  apparatus  for 
updating  the  skills  of  manufacturing 
personnel.  The  systems  engineering  process 
and  documentation  can  be  used  to  identify 
facility  and  personnel  requirements  for  this 
type  of  production  support  facility. 

18.7  PRODUCTION  STRATEGY/PLAN 
DEVELOPMENT 

A  production  strategy  is  developed  as 
part  of  the  overall  program  acquisition 
strategy.  This  strategy  is  a  comprehensive 
assessment  of  production  issues  that  forms 
the  foundation  for  a  formal  Production  Plan. 
The  Production  Plan  is  prepared  by 
contractor(s)  during  C/E  and  is  presented  as 
a  completed  document  during  FSD.  DoD 
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4245.7-M  [2],  also  provides  useful  supporting 
guidance  for  developing  production  planning. 

Production  planning  review  is  an  integral 
part  of  the  overall  acquisition  review 
process.  An  acquisition  may  not  proceed 
into  production  until  it  is  determined  that 
the  principal  contractors  have  the  physical, 
financial,  and  managerial  capacities  to  meet 
the  cost  and  schedule  commitments  of  the 
proposed  procurement.  An  assessment  is 
made  of  contractors’  capabilities  to  meet 
surge  (peacetime)  and  mobilization  (declared 
national  emergency)  requirements,  and  the 
contractors’  commitment  to  participate  in  the 
DoD  industrial  preparedness  production 
planning  program  under  DoDD  4005.1  (8]. 
Competition,  value  engineering,  tailoring  of 
specifications  and  standards,  design  to  cost, 
cost  benefit  and  trade-off  assessments, 
preplanned  product  improvements, 
multi-year  procurements,  industrial 
modernization  incentives,  and  other 
techniques  are  used  to  reduce  production, 
operating,  and  support  costs. 
Standardization,  commonality,  and 
interchangeability  must  be  promoted 
throughout  the  acquisition  cycle  to  reduce 
lead  time  and  life  cycle  cost. 

Production  management  planning  and 
implementation  also  includes  provisions  for 
measuring  progress  toward  design  to  cost 
and  life  cycle  cost  commitments. 

18.8  PRODUCIBILITY  ENGINEERING 
AND  PLANNING  (PEP) 

The  term  "producibility  engineering  and 
planning"  as  used  in  DoD  is  identical  to  the 
term  "production  planning"  in  the  academic 


and  industrial  worlds.  PEP  includes  all  those 
design  activities  and  disciplines  necessary  to 
design  a  product  that  is  producible,  design 
the  processes  and  tooling,  set  up  the 
manufacturing  facility,  and  prove  the 
processes  and  facilities,  before  entering 
production. 

DoD  policy  states  that  a  comprehensive 
PEP  program  is  requisite  to  entering  FSD. 
The  PEP  program  begins  as  an  integral  part 
of  the  design  process  and  is  conducted 
throughout  FSD.  It  contains  specific  tasks, 
measurable  goals,  and  a  system  for 
contractor  accountability. 

The  contractor’s  System  Engineering 
Management  Plan  (SEMP)  should  define 
how  the  PEP  program  will  integrate  into  the 
overall  systems  engineering  process. 
Normally,  this  will  be  an  extension  of  the 
procedures  used  during  C/E  and  D/V  to 
integrate  producibility  "design-for" 
requirements  into  the  engineering 
management  process. 

The  PEP  policy  states  that  PMs  are  to 
make  assessments  of  production  risk 
throughout  the  acquisition  process.  These 
assessments  are  formalized  by  reviewing  the 
design  process  through  the  established  design 
review  and  audit  mechanism,  industrial 
resource  analysis,  and  Production  Readiness 
Reviews  (PRRs).  Normally,  production  risk 
assessment  would  be  an  integral  part  of  the 
overall  risk  management  approach.  This 
relationship  should  also  be  detailed  in  the 
contractor’s  SEMP  as  well  as  PEP  planning. 

The  PEP  program  extends  throughout 
the  life  cycle.  It  includes  actions  required  to 
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Figure  18-5 

Production  Strategy/Plan  Development  Outline 


1.  Program  Production  Objective 

a.  Requirements 

b.  Additional  Program  Needs 

2.  Production  Facility  Requirements 

a.  Existing  Industrial  Base 

b.  Government -Furni shed  Facilities 

c.  Contractor  Facilities/Capital  Investment 

3.  Production  Program  Phasing 

a.  Production/Delivery  Schedule 

b.  Economic  Production  Rate/Planned  Production  Rate 

c .  Built -Up  Rate 

4 .  Production  Risk  Analysis 

a.  Management 

b.  Materials/Purchased  Parts 

c.  Faci 1 i t ies /Equipment 

d.  Labor 

e.  Design  Changes 

f.  Producibi  1  i  ty 

5.  Producibi 1 i ty  Engineering  and  Planning 

a.  Design  Producibi 1 itv 

b .  PEP 

(1)  Funding  Controls 

(2)  Objective 

(3)  Work  Scope 

(4)  Allocation  of  Human  Resources 

(5)  Data  Items 

(6)  Time/Fund  Allotment 

6.  Contractual  Consideration 

a.  Statement  of  Work 

b.  Incentives 

c.  Consistency  Within  and  Among  Contracts 

7.  Government  Support  to  Contractor 

a.  Government-Furnished 

b.  Defense  Materials  System/Defense  Priorities  System 


8.  Production  Management  Scheme 

a.  Program/Matrix  Staffing 

b.  Plant  Representative  Office 

c.  DoD  Research  and  Development  Laboratories 

d.  Consultant  Support 

e.  Other  Government  Team  Participation 
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maintain  a  capability  to  produce  material  for 
equipment  operation  and  maintenance  after 
the  Production  Plan  is  complete.  The 
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CHAPTER  19 


INTEGRATED  LOGISTIC  SUPPORT  (ILS) 


19.1  INTRODUCTION 

The  purpose  of  this  chapter  is  to 
address  the  role  of  integrated  logistics 
support  (ILS)  within  the  context  of  the 
systems  engineering  process.  The  discussion 
centers  on  the  key  linkages  between  systems 
engineering  and  components  of  an  effective 
ILS  program.  The  reader  is  referred  to 
reference  [1]  for  more  detailed  treatment  of 
the  ILS  process  as  it  applies  to  the 
acquisition  life  cycle. 

System  readiness  is  a  primary 
objective  of  the  acquisition  process. 
Department  of  Defense  (DoD)  policy 
requires  that  resources  to  achieve  readiness 
;eceive  the  same  emphasis  as  those  required 
to  achieve  schedule  and  performance 
objectives  (DoDD  5000.1  [2]).  Such 

resources  include  those  necessary  to  design 
desirable  support  characteristics  into  material 
systems,  as  well  as  those  necessary  to  plan, 
develop,  acquire,  and  evaluate  the  support. 

DoDD  5000.39  [3]  emphasizes  early 
identification  of  supportability  design 
requirements  through  integration  with  the 
mainstream  engineering  effort.  One  way  to 
achieve  this  is  to  establish  a  rigorous  formal 
relationship  between  the  ILS  process  arid  the 
systems  engineering  process.  An  ILS 
program  success  hinges  on  how  the  readiness 
and  supportability  characteristics  are 
designed  into  the  system.  These 
characteristics  must  be  designed  in  early, 


during  Concept  Exploration/  Definition 
(C/E)  and  Concept  Demonstration/ 
Validation  (D/V),  and  continuously  re¬ 
evaluated  through  Full  Scale  Development 
(FSD).  They  must  be  considered  in 
performing  functional  and  trade-off  analyses. 
The  systems  engineering  process  provides  a 
framework  for  enabling  the  effective 
acquisition  of  a  supportable  system. 

Figure  19-1  illustrates  the  analytic  and 
decision  making  process  involved  in  the 
application  of  systems  engineering  to 
acquisition  management.  The  example 
shown  is  for  the  C/E  and  D/V  phases. 
Within  the  framework  of  this  process, 
government  operational  needs  are  analyzed; 
the  various  design  concepts  are  synthesized, 
evaluated,  and  optimized  in  trade  studies; 
and  the  "best"  design  is  selected. 

The  upper  portion  of  Figure  19-1 
portrays  those  systems  engineering  efforts 
and  activities  that  define  the  requirements 
for  prime  item  equipment  and  associated 
software.  The  lower  portion  of  Figure  19-1 
contains  efforts  and  activities  that  define  the 
related  logistic  support  requirements. 
Attainable  supportability  characteristics  are 
defined  throughout  the  design  process  using 
design  trade-off  efforts  involving  all  product 
design  and  support  disciplines,  including 
reliability  and  maintainability  (R&M).  The 
contractor  must  give  equal  emphasis  during 
requirements  identification  and  allocation  to 
aiialysis  of  support  functions  using  tools  such 
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Figure  19-1 

ILS  and  the  Systems  Engineering  Process 


as  the  functional  flow  block  diagram  (FFBD) 
and  requirements  allocation  sheet  (RAS). 

To  achieve  the  necessary  balance  of  ILS 
factors  within  the  systems  engineering 
process,  the  contractor  must  define  trade-  off 
and  decision  criteria  that  adequately  address 


support  requirements.  A  balanced 
integration  of  logistic  support  requirements 
into  the  systems  engineering  process  will 
achieve  the  following  objectives: 

a.  Accomplish  readiness  objectives  that 
will  be  challenging  but  attainable 


Typical  Contractor  Systems  Engineering 
Linkages  with  Logistic  Support  Elements 
(for  Aircraft  Development  Programs) 
Figure  19-2 


PHASES  IN  THE  LIFE  CYCLE 


System  Acquisition  Management 
Figure  19-3 
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b.  Accomplish  realistic  R&M 
requirements  to  achieve  these  objectives 

c.  Identify  support  and  manpower  drivers 

d.  Assign  appropriate  priority  to  ILS 
element  requirements  in  system  design 
trade-offs. 

19.2  1LS-SYSTEMS  ENGINEERING 
MANAGEMENT  INTERACTION 

Figure  19-2  illustrates  typical  areas  of 
management  interaction  between  systems 
engineering  and  ILS  program  elements  for 
an  aircraft  development  program. 

This  figure  shows  a  broad  array  of 
logistics  related  functional  disciplines  in 
organizational  cells  on  the  left  side  of  the 
linkage  diagram  and  illustrates  the 
complexity  of  integrating  support  into  the 
design  process  of  large  programs.  These 
linkages  must  be  formally  addressed  in  the 
contractor’s  System  Engineering 
Management  Plan  (SEMP)  and  Integrated 
Logistic  Support  Plan  (ILSP).  The  ILS  role 
in  relation  to  these  interactions  is  discussed 
below. 

Systems  engineering  supportability 
characteristic  outputs  are  developed  by 
discipline  specialists  participating  in  the 
systems  engineering  process  prior  to  FSD. 
Functional  and  allocated  baselines  are 
developed  during  C/E  and  D/V,  respectively. 
The  successful  integration  of  ILS  into  system 
design  is  partially  demonstrated  by  the 
extent  of  effective  supportability 
characteristics  and  requirements  in  system 
specifications  (Type  A)  and  development 
specifications  (Type  B).  Figure  19-3  displays 
the  phasing  of  the  functional,  allocated,  and 


product  baselines  and  their  related 
specifications  with  respect  to  other  system 
acquisition  management  milestones.  The 
dotted  lines  in  Figure  19-3  portray  the 
period  of  documentation  review,  while  the 
solid  lines  portray  continued  use  under 
government  configuration  management.  The 
format  of  the  system  specification  (type  A) 
has  provisions  for  identification  of 
supportability  characteristics  (R&M)  and 
logistics  concept  requirements  (e.g., 
maintenance,  supply,  and  facilities).  The 
development  specifications  (type  B)  contains 
requirements  for  major  configuration  items 
(CIs),  components,  and  software.  These 
specifications  (Types  A  and  B)  control  the 
engineering  design  activities  (refer  to  the 
upper  right  portion  of  Figure  19-2)  during 
FSD. 

The  product  baseline  release  in  Figure 
19-2  provides  detailed  design  documentation 
for  the  transition  to  production.  Timely 
release  of  the  major  configuration  items  and 
their  support  and  training  equipment  designs 
is  required  for  scheduling  logistic  activities 
such  as  preparation  of  final  technical 
manuals,  preparation  and  processing  of 
provisioning  documentation,  and 
development  of  packaging  requirements. 

Material  release  refers  to  the  decision  to 
proceed  with  deployment  of  the  first  system 
in  its  military  role.  This  event  signifies  that 
all  support  has  been  acquired  and  can  be 
provided  concurrently,  or  prior  t  this  initial 
deployment. 

Field  data  and  experience  provide  the 
means  of  assessing  supportability  and 
attained  readiness,  instituting  required 
improvements,  and  updating  the  ILS 
elements. 
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193  COMPUTER-AIDED  ACQUISITION 
AND  LOGISTIC  SUPPORT  (CALS) 

Widespread  industry  use  of  computer 
aided  design  and  engineering  (CAD/CAE) 
has  created  a  new  environment  where 
product  description  data  is  becoming 
available  in  digital  form  to  support  a  wide 
range  of  DoD  and  industry  applications. 
CALS  is  a  DoD  and  industry  program  to 
enable  and  accelerate  the  use  and 
integration  of  this  digital  technical 
inform?/.  <  tr.pon  system  acquisition, 
design.  manufacture  and 
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CALS  will  transition  from  current  paper 
intensive  mode  to  a  highly  automated  and 
integrated  mode  of  operation  ,  thereby 
substantially  improving  productivity  and 
quality.  The  Deputy  Secretary  of  Defense 
launched  the  DoD  CALS  program  in  1985, 
with  the  goal  that  by  1990  new  weapon 
systems  would  acquire  technical  data  in 
digital  form,  or  obtain  government  access  to 
contractor  integrated  data  bases  in  lieu  of 
paper  deliverables.  The  potential  exists  for 
substantial  quality  improvements  and 
reductions  in  acquisition  and  support  costs 
through  CALS. 

To  achieve  these  benefits,  a  phased 
CALS  strategy  has  been  planned  by  a  team 
consisting  of  the  Office  of  the  Secretary  of 
Defense  (OSD),  the  Services,  and  industry. 
Phase  I  will  replace  paper  documentation 
transfers  with  digital  file  exchanges  and 
begin  process  integration.  This  will  be 
implemented  between  now  and  the  early 
1990’s.  In  parallel,  technology  is  being 
developed  for  Phase  II  which  involves 
substantial  integration  of  current  processes 
to  take  advantage  of  a  shared  data  base 


environment  in  the  early  1990s  and  beyond. 
The  main  roles  od  DoD  in  both  phase  are 
1)  to  accelerate  the  development  and  test  of 
interchange  and  data  access  standards,  2)  to 
fund  demonstrations  and  technology 
development  in  high  risk  areas,  3)  to 
encourage  industry  investment  in  integrated 
processes  by  establishing  contract 
requirements  and  incentives,  and  4)  to 
implement  CALS  capabilities  in  DoD’s  own 
extensive  automated  systems. 

The  CALS  program  has  made  significant 
progress.  Technical  information  exchange 
standards  have  been  publihed,  a  standards 
application  testing  program  has  begun,  and 
the  incorporation  of  CALS  concepts  into 
DoD  and  industrial  infrastructures  is 
underway.  Advance  technology  reswarch  and 
development  to  meet  long  term  CALS 
requirements  is  being  accelerated  through 
close  DoD  and  industry  collaboration. 

CALS  planning  efforts  are  concentrating 
on  the  orderly  insertion  of  technological 
advances  in  digital  technical  data 
management  and  use  into  the  existing 
defense  acquisition  and  logistic  support 
processes.  CALS  provides  a  unique 
opportunity  to  achieve  major  productivity 
and  quality  improvements  through  carefully 
planned  and  managed  investment  by  both 
DoD  and  industry.  Initially,  the  changes  will 
be  gradual,  as  building  blocks  are  put  into 
place  and  specific  portions  of  the  weapon 
system  life  cycle  are  enhanced.  Benefits  will 
begin  to  accrue  as  productivity  improves  in 
both  DoD  and  industry.  CALS 
implementation  will  result  in  lower  weapon 
system  life  cycle  costs,  shortened  acquisition 
times,  and  improvements  in  reliability  and 
maintainability.  Most  importantly,  the  ability 
of  the  Defense  components  to  perform  their 
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Figure  19-4 

Basic  and  Special  Purpose  Systems 
Engineering  Documentation 
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assigned  missions  will  continue  during  this 
technology  insertion  process. 

19.4  SYSTEMS  ENGINEERING  AND 
LOGISTIC  SUPPORT  ANALYSIS 

It  is  the  resposibility  of  the  government 
Program  Manager  (PM)  to  coordinate  the 
communication  and  planning  of  work 
between  the  design  engineers  and  the 
logistics  managers.  The  ILS  Manager’s  LSA 
Plan  should  identify  the  scope  of  analytical 
effort  for  each  acquisition  phase,  and  must 
be  coordinated  with  the  contractor's  SEMi  ' 
Together  these  documents  dt'  i 
detailed  relationship  of  both  the  anaiy.- ■•“>• 
data  developed  under  MIL-STT  ’  ">66-1/.  ;  \ 
and  MIL-STD-499A  [5].  A  general  area  of 
concern  is  the  non-duplication  of  analysis 
and  data,  and  the  requirement  for 
traceability  between  systems  engineering  and 
LSA  data  elements.  It  should  be  noted  that 
analysis  of  logistic  support  functions  has 
always  been  an  integral  pan  of  the  systems 
engineering  process  described  by  military 
specifications  and  standards. 
MIL-STD-1388-1A  [4]  provides  a 

contemporary  focus  on  specific  requirements. 

Figure  19-4  illustrates  the  basic 
documentation  used  in  the  systems 
engineering  process.  It  can  be  seen  that 
many  of  these  documentation  Items  define 
and  describe  logistic  elements.  The  LSA 
Record  (LSAR)  provides  an  expansion  of 
the  maintenance  analysis  defined  on  the  End 
Item  Maintenance  Sheet  (EIMS).  Figure 
19-5  shows  the  relationship  between  systems 
engineering  documentation  and  the  LSA 
tasks  identified  in  MIL-STD- 1388-1 A  [4]  .The 
right-hand  section  of  the  matrix  illustrates 
the  principal  interface  with  the  systems 
engineering  documentation  described  in 


Figure  19-4  and  MIL-STD-499A  [5].  Each 
interfacing  area  should  be  described  in  detail 
in  the  contractor  LSA  and  SEMP  planning 
to  avoid  duplication  of  effort  and  provide 
traceability  and  correlation  of  system 
requirements. 

MIL-STD- 1388- 2A  [6]  is  presently 
undergoing  revision  as  MIL-STD- 1388-2B. 
Under  this  revision,  the  LSA  Records  (A-H 
and  J)  will  be  deleted  and  the  LSA  Master 
tables  will  be  replaced  by  relational  tables 
formats.  The  number  of  reports  will  be 
reduced  from  80  to  21.  These  remaining  21 
teports  are  product-oriented  reports  and  not 
analysis  reports.  The  accompanying  Joint 
Services  LSAR  automated  data  processing 
(ADP)  system  is  being  developed  as  a 
relational  data  base  platform  which  will 
provide  the  following  advantages:  1)  the 
LSAR  ADP  system  will  be  easier  to 
change/update,.  2)  the  "hooks"  for  using 
interactive  systems  engineering  tools  (e.g., 
level  of  repair  modeling)  will  be  available,  3) 
one  time  storage  of  data  elements  will 
improve  the  consistency  and  quality  of 
products,  4)ad  hoc  report  capability,  and  5) 
online  access  and  review  of  LSAR  data. 
Front  end  data  entry  screens  and  report 
generation  capabilities  will  also  be  available 
as  part  of  the  LSAR  software.  This  software 
package  will  be  exportable  to  industry  and 
government  users. 

19.5  IMPACT  OF  R&M  ON  ILS 

R&M  parameters  are  the  most  effective 
logistics  engineering  tools  for  influencing  and 
interacting  with  the  systems  engineering 
process.  Establishment  of  effective  R&M 
requirements  for  the  total  system  and  the 
allocation  of  tose  requirements  to  lower  level 
components  are  a  vital  influence  on  mission 
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Figure  19-5(2) 

LSA/Systems  Engineering  Documentation  Interface 


19-10 


success  and  operation  and  support  (O&S) 
cost.  R&M  parameters  listed  in  DoDD 
5000.40  [7]  and  summarized  in  Figure  19-6, 
include  the  following:  mission  success, 
readiness,  maintenance  manpower 
requirements  and  costs,  and  logistic  support 
cost. 

Mission  success  is  greatly  influenced  by 
mission  reliability  (mean  time  between 
critical  failures  that  impact  the  mission)  and 


mission  maintainability  (mean  time  to  restore 
functions  during  the  mission).  Readiness  is 
partially  determined  by  mean  time  between 
downing  events  and  mean  time  to  restore 
the  system.  Maintenance  manpower 
requirements  and  costs  are  affected  by  the 
time  between  the  manhours  to  perform 
maintenance  actions.  Logistic  support  costs 
related  to  parts  are  determined  by  the  mean 
time  between  removal  of  repairables  and 
consumables,  and  the  total  of  all  costs  to 


System  R&M  Parameters 
Figure  19-6 
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remove,  replace,  transport,  and  repair 
components  at  all  levels  of  maintenance. 

The  ILS  program  must  ensure  that  the 
R&M  parameters  are  consistent  with 
planned  peacetime  and  wartime  operational 
environments/scenarios,  and  the  support  that 
will  be  provided  under  these  conditions. 
Failure  to  fully  account  for  the  effects  of 
item  design,  quality,  operation,  maintenance, 
and  repair  can  lead  to  a  substantial  shortfall 
in  operational  performance  and  an  overrun 
of  logistic  support  costs. 

R&M  considerations,  as  other  ILS 
considerations,  are  incorporated  into  system 
design  through  the  systems  engineering 
process.  In  particular,  the  synthesis  and 
trade-off  analysis  methodologies  described  in 
Chapters  7  and  8  of  this  guide  ensure  that 
ILS  parameters  are  incorporated  along  with 
other  technical  and  program  requirements  to 
produce  a  balanced  design. 

19,6  SUPPORT  SYSTEM  DESIGN 
AND  SYSTEMS  ENGINEERING 

The  support  system  design  functions 
usually  include  the  design  of  automatic  and 
non-automatic  test  and  support  equipment, 
simulators,  training  equipment,  mobile 
maintenance  trainers,  analysis  of 
maintenance  and  repair  facility  requirements, 
and  packaging  and  transportation  studies. 
Using  LSA  and  systems  engineering 
techniques,  the  ILS  program  should:  1) 
Integrate  system  performance  and  support 
requirements  and  incorporate  support 
parameters  in  system  specifications,  2) 
Identify  ILS  discipline  design  requirements 
for  configuration  items  early  and  refine  them 
throughout  the  life  cycle,  3)  Analyze 
government-furnished  equipment  (GFE) 


support  system  items  and  integrate  these 
items  into  the  total  system  design,  4) 
Emphasize  ILS  requirements  relative  to 
hardware,  firmware,  and  software  interface 
design  considerations,  and  5)  Include 
procedures  for  determining  the  effect  of 
support  system  design  items  in  the 
configuration  management  program  to 
provide  total  system  consideration  of 
proposed  changes. 

19.7  COMPUTER  SOFTWARE 
SUPPORTABILITY 

Software  design  and  support 
considerations  are  important  to  the  success 
of  the  ILS  program.  ILS  planning  should 
incorporate  requirements  of 
DoD-STD-2167A  [8]  for  ensuring  the 
supportability  of  all  computer  software.  This 
will  provide  a  software  management  system 
that  parallels  Jhe  hardware  requirement 
system.  DoD-STD-2167A  [8]  requirements 
for  software  are  explained  in  detail  in 
Chapter  20  of  this  guide.  In  conjunction 
with  the  approved  SEMP  under 
MIL-STD-499A  [5],  DoD-STD-2167A  [8] 
provides  a  controlled  and  structured  software 
development  process  involving  software 
design  specifications,  design  reviews, 
milestones,  documentation,  configuration 
control  and  identification,  validation,  and 
verification. 

In  addition,  ILS  planning  should  assure 
that  either  the  ILSP  or  the  Computer 
Resources  Life  Cycle  Management  Plan 
(CRLCMP)  adequately  address  support 
planning  for  user  programmable  firmware 
and  software.  This  planning  should  highlight 
documentation,  training,  support  equipment, 
and  facility  requirements  for  software  and  be 
consistent  with  systems  engineering,  software. 
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Due  to  the  increasingly  important  role  of 
software,  special  attention  should  be  given  to 
identifying  and  evaluating  software 
maintenance  requirements  during  systems 
analysis  and  planning  for  adequate  hardware 
and  services  support  during  the  operation 
phase  of  the  program. 

19.8  SUMMARY 

The  systems  engineering  process 
produces  a  balanced  design  that  will  reflect 
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CHAPTER  20 


SOFTWARE  DEVELOPMENT  PROCESS 


20.1  INTRODUCTION 

The  development  of  a  weapon  system 
requires  integrating  technical,  administrative, 
and  management  disciplines  into  a  cohesive, 
well-planned,  and  rigorously  controlled 
process.  As  a  critical  component  of  a 
weapon  system,  software  must  be  developed 
under  a  similarly  disciplined  engineering 
process.  In  his  book  "Software  Engineering 
Concepts"  [1],  Richard  Fairley  defines 
software  engineering  as  "the  technological  and 
managerial  discipline  concerned  with  systematic 
production  and  maintenance  of  software 
products  that  are  developed  and  modified  on 
time  and  within  cost  estimates ". 

Barry  Boehm  [2]  defines  software 
engineering  as  a  discipline  that  " involves  the 
practical  application  of  scientific  knowledge  to 
the  design  and  construction  of  computer 
programs  and  the  associated  documentation 
required  to  develop,  operate,  and  maintain 
them ", 

The  main  point  is  that  the  software 
development  process  must  be  scientific  and 
disciplined.  This  is  not  different  from  the 
hardware  development  process.  As  with 
hardware,  the  goal  of  the  software 
development  process  is  to  consistently 
produce  a  quality  product,  within  schedule 
and  cost. 


With  the  publication  of 
DOD-STD-2167A  [3],  the  DoD  took  the 
first  step  toward  a  standardized,  systems 
engineering  approach  to  software 
development.  This  standard  is  supported  by 
other  military  documents  and  describes  a 
standard  process  and  documentation  for 
computer  software  development.  To  use  this 
standard  effectively,  the  systems  engineer 
must  have  a  thorough  understanding  of  the 
system  being  developed;  particularly  the 
overall  system  requirements  and  constraints. 
Requirements  must  be  defined  early  through 
trade  studies  and  prototyping.  Traceability  of 
requirements  must  be  maintained  throughout 
the  acquisition  life  cycle  and  any 
requirement  that  cannot  be  traced  up  to  a 
higher  requirement  should  be  modified  or 
eliminated. 

The  material  presented  in  this  chapter 
was  extracted  from  the  DSMC  "Mission 
Critical  Computer  Resources  Management 
Guide"  [4]  and  modified  accordingly.  It  will 
describe  activities  that  occur  in  a  "typical" 
program.  The  reader  should  understand 
that  real  programs  seldom  actually  follow 
this  "typical"  profile.  Phases  can  occur 
concurrently,  they  can  be  bypassed 
altogether,  protracted,  or  condensed  to 
satisfy  the  needs  of  the  overall  program 
objectives.  The  point  to  understand  is  that 
although  the  process  is  somewhat  constant, 


20-1 


its  chronological  occurrence  is  not  fixed. 
The  following  sections  describes  the  classical 
approach  to  software  development. 

20.2  SUMMARY  OF  DEVELOPMENT 
ACTIVITIES 

Figure  20-1  presents  an  overview  of 
the  development  activities  of  an  integrated 
software  and  hardware  system  as  reflected  in 
DOD-STD-2167A  [3].  ' 

All  weapon  system  development 
programs  begin  with  a  determination  of 
system  level  requirements.  These  activities 
occur  during  the  Concept  Exploration/ 
Definition  (C/E)  and  the  Concept 
Demonstration/  Validation  (D/V)  phases  of 
the  acquisition  cycle. 


The  Systems  Requirements  Review 
(SRR)  may  be  held  after  the  initial 
determination  of  system  functions  (functional 
analysis)  and  the  preliminary  allocation  of 
these  functions  io  configuration  items.  The 
SRR  provides  an  opportunity  for  an  initial 
insight  into  the  developer’s  direction, 
progress  and  convergence  on  a  system 
configuration. 

The  System  Design  Review  (SDR)  is  a 
review  of  the  overall  system  requirements  in 
order  to  establish  the  system  functional 
baseline  documented  by  the  system 
specification.  The  functional  baseline  should 
allocate  requirements  to  both  hardware  and 
software  configuration  items. 

The  development  of  both  hardware 


Figure  20-1  Software/  Hardware  Development 
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and  software  can  begin  once  the  functional 
baseline  is  established.  These  activities  occur 
in  the  Full  Scale  Development  (FSD)  phase 
and  are  monitored  through  informal  and 
formal  reviews  and  audits  as  described  in 
MIL-STD-1521B  [5j.  The  allocated  baseline 
for  software  should  be  established  at  the 
Software  Specification  Review  (SSR).  For 
hardware  the  allocated  baseline  is  normally 
established  at  the  Preliminary  Design  Review 
(PDR);  but  no  later  than  the  Critical  Design 
Review  (CDR). 

Building  of  the  developmental  products 
can  start  once  the  design  effort  is  completed. 
For  hardware  this  building  effort  is  called 
fabrication  and  for  software  it  is  called 
coding  and  testing.  Testing  is  further 
subdivided  into  Computer  Software  Unit 
(CSU)  testing  and  Computer  Software 
Component  (CSC)  integration  and  testing. 
After  the  items  are  built,  formalized  testing 
should  take  place  in  accordance  with 
approved  test  plans  and  procedures.  A  Test 
Readiness  Review  (TRR)  will  be  conducted 
by  the  government  in  order  to  determine  the 
developer’s  readiness  to  perform  formalized 
acceptance  testing.  Completion  of  software 
testing  will  lead  to  system  integration  and 
testing.  Both  Functional  Configuration 
Audits  (FCA)  and  Physical  Configuration 
Audits  (PCA)  will  be  conducted  on  both 
hardware  and  software  configuration  items  in 
order  to  establish  the  respective  product 
baselines.  After  a  Formal  Qualification 
Review  (FQR)  at  the  system  level,  the 
integrated  system  will  be  turned  over  to  the 
government  for  operational  testing  as 
defined  in  the  system’s  Test  and  Evaluation 
Master  Plan  (TEMP).  Successful  completion 
of  this  testing  indicates  that  the  product  is 
fully  defined  and  ready  to  be  manufactured. 


For  hardware,  the  production  line  would 
begin  to  assemble  carbon  copy  items.  For 
software,  turning  out  copies  is  a  trivial 
process  since  the  product  is  complete  and 
needs  only  to  be  duplicated  on  the  required 
media  for  transfer  to  the  target  system 
computer. 

203  SYSTEM  REQUIREMENTS 

ANALYSIS/DESIGN 

Figure  20-2  depicts  the  activities  and 
products  associated  with  the  C/E  and  D/V 
phases.  The  C/E  and  D/V  phase  activities 
are  system  oriented  to: 

a.  Define  overall  project  objectives 

b.  Determine  project  feasibility 

c.  Develop  acquisition/development 
strategy 

d.  Establish  resource  cost  and  schedule 

e.  Define  hardware/software 
interrelationships 

f.  Define  technical  and  business 
functions  and  performance. 

The  first  step  in  any  system 
development  is  to  generate  the  system  level 
requirements  and  reflect  them  in  a  System/ 
Segment  Specification  (SSS)  (Type  A 
Specification).  It  doesn’t  make  any  difference 
whether  it  is  a  hardware  only,  a  software 
only,  or  a  hardware  and  software  system;  the 
most  important  and  critical  aspect  of  weapon 
system  development  is  to  "nail  down"  the 
system  requirements.  These  requirements 
must  first  be  finalized  at  the  functional  level, 
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before  being  allocated  to  hardware  and 
software. 

The  requirements  are  nailed  down 
through  a  series  of  engineering  studies  and 
trade-offs.  These  studies  include: 

a.  Requirements  Refinement  -  The 
overall  system  requirements,  including 
constraints,  should  be  examined  to  identify 
the  factors  that  drive  requirements  for 
computer  resources.  These  factors  may 
include  system  interfaces,  interoperability, 
communication  functions,  personnel 
functions,  the  anticipated  level  and  urgency 
of  change,  and  requirements  for  reliability 
and  responsive  support. 

b.  Operational  Concept  Analysis  - 

The  operational  concept  should  be  analyzed 


in  order  to  determine  the  role  of  computer 
resources.  Particular  attention  is  paid  to 
requirements  for  mission  preparation, 
operator  interface,  control  functions,  and 
mission  analysis. 

c.  Trade-off  and  Optimization  -  The 

effects  of  system  constraints  such  as  the 
operations  concept,  the  support  concept, 
performance  requirements,  logistics, 
availability  and  maturity  of  technology,  and 
limitations  on  cost,  schedule,  and  resources 
are  determined.  Alternative  computer 
resources  approaches  are  studied  to: 

1)  meet  operational, 
interoperability,  and  support  requirements 

2)  determine  how  the  system 
requirements  of  reliability  and  maintainability 
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Figure  20-2  System  Requirements  Analysis 
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(R  &  M)  will  be  satisfied 

3)  determine  how  requirements  for 
system  security  will  be  met. 

A  determination  will  also  be  made 
regarding  the  suitability  of  standard 
computer  languages  and  instruction  set 
architectures. 

d.  Risk  -  For  each  approach,  the 
risks  associated  with  computer  resources  are 
evaluated.  Typical  risk  areas  include 
compiler  maturity,  availability  and  maturity 
of  the  software  support  tools,  loosely  defined 
or  incomplete  interface  definitions,  and  lack 
of  adequate  computer  memory  or 
throughput  capability. 

203.1  System  Design 

System  design  begins  on  or  about  the 
time  of  the  SRR.  The  major  function  of 
system  design  is  to  establish  the  functional 
aseline  of  the  system  by  updating  and 
approving  the  system  specification  and  the 
operational  concept;  by  developing  the  initial 
subsystem/segment  designs;  and  by  further 
refining  the  systems  engineering  planning 
activities  to  be  employed  during  system’s 
development.  Typical  products  are: 

a.  System  Specification 

b.  System/Segment  Designs 

c.  Configur?  Management  Plan 

(CMP) 

d.  Computer  Resources  Life  Cycle 
Management  Plan  (CRLCMP) 


e.  Preliminary  Software  Requirements 
Specification  (SRS); 

f.  Preliminary  Interface  Requirements 
Specification  (IRS); 

20.4  SOFTWARE  DEVELOPMENT 

Before  undertaking  a  discussion  of 
software  development  it  will  be  necessary  to 
present  the  following  definitions: 

a.  Computer  Software  Confieuration 
Item  (CSCD  -  A  configuration  item  for 
computer  software. 

b.  Computer  Software  Component 
(CSCI  -  A  distinct  part  of  a  computer 
software  configuration  item  (CSCI).  CSCs 
may  be  further  decomposed  into  other  CSCs 
and  Computer  Software  Units. 

c.  Computer  Software  Unit  -  An 
element  specified  in  the  design  of  a 
Computer  Software  Component  (CSC)  that 
is  separately  testable.  A  CSU  is  the  lowest 
level  of  software  decomposition. 

Weapon  system  software  is  partitioned 
into  CSCIs  based  on  the  program  office’s 
management  strategy.  Each  CSCI  is 
managed  individually  and  follows  its  own 
development  cycle.  The  software 
development  cycle  is  defined  in 
DOD-STD-2167A  [3]  and  consists  of  eight 
major  activities:  1)  Systems  Requirements 
Analysis/Design,  2)  Software  Requirements 
Analysis,  3)  Preliminary  Design,  4)  Detailed 
Design,  5)  Coding  and  CSU  Testing,  6)  CSC 
Integration  and  Testing,  7)  CSCI  Testing, and 
8)  Systems  Integration  and  Testing. 
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These  steps  typically  occur  during  FSD, 
although  they  may  occur  one  or  more  times 
during  each  of  the  system  life  cycle  phases 
[6].  This  is  especially  true  if  software 
prototyping  is  performed  during  the  D/V 
Phase.  The  steps  are  not  simply  linear  since 
software  development  is  iterative  in  nature 
and  any  step  may  be  repeated  many  times 
during  the  course  of  system  development. 
For  example,  problems  discovered  during 
software  integration  and  testing  may  force 
the  software  designers  to  go  back  and  redo 
the  Software  Requirements  Analysis  and  all 
the  subsequent  steps. 

Managing  software  is  very  similar  to 
managing  hardware;  both  require  discipline 
and  control  in  order  to  succeed.  An 
important  part  of  the  control  process  is  the 
formal  determination  of  whether  or  not  the 
developer  is  ready  to  proceed  to  the  next 
step.  This  is  usually  determined  through  a 
series  of  design  reviews  and  audits. 
Software  reviews  and  audits  can  occur  in 
conjunction  with  hardware  reviews;  but  they 
do  not  necessarily  have  to.  It  is  important 
that  appropriate  system  level  reviews  be 
held  at  strategic  intervals.  This  will  focus 
everyone’s  (hardware  and  software 
personnel)  attention  on  system  design  and 
leads  to  timely  baselines  for  the  hardware, 
the  software,  and  all  the  interfaces.  Software 
development  has  two  major  reviews  that  are 
separate  from  hardware  reviews:  the 
Software  Specification  Review  (SSR)  and  the 
Test  Readiness  Review  (TRR). 

The  SSR  is  a  formal  review  of  a  CSCI’s 
requirements  as  specified  in  the  software 
specifications  .  A  collective  SSR  for  a  group 
of  configuration  items  (CIs),  addressing  each 


configuration  item  individually,  may  be  held 
when  such  an  approach  is  advantageous  to 
the  government.  Its  purpose  is  to  establish 
the  allocated  baseline  for  preliminary  CSCI 
design  by  demonstrating  to  the  government 
the  adequacy  of  the  software  specification. 

The  TRR  is  a  formal  review  of  the 
contractor’s  readiness  to  begin  formal  CSCI 
testing.  It  is  conducted  after  software  test 
procedures  are  available  and  CSC 
integration  testing  is  complete.  The  purpose 
of  the  TRR  is  to  determine  whether  the 
contractor  is  ready  to  begin  formal  CSCI 
testing  that  can  be  witnessed  by  the 
government.  A  technical  understanding  must 
be  reached  on  the  informal  test  results,  and 
on  the  validity  and  the  degree  of 
completeness  of  such  documents  as  an 
operator’s  manual,  a  user’s  manual,  and  a 
computer  programmer’s  manual. 

20.4.1  Software  Requirements  Analysis 

The  first  step  in  the  software 
development  cycle  is  the  Software 
Requirements  Analysis  (Figure  20-3).  The 
purpose  of  the  Software  Requirements 
Analysis  is  to  establish  detailed  functional, 
performance,  interface,  and  qualification 
requirements  for  each  CSCI  based  on  the 
System  Specification.  The  means  of  testing 
and  examining  the  software  are  also 
identified.  During  requirements  analysis, 
prototype  versions  of  high  risk  areas,  user 
interfaces,  and/or  systems  skeletons  may  be 
partially  designed  and  coded.  Prototyping  is 
an  excellent  tool  for  performing 
requirements  analysis. 

The  developer  should  also  identify 
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support  tools  and  resources,  and  establish 
timing  and  sizing  estimates.  The  Program 
Manager  must  ensure  that  all  software 
requirements,  as  reflected  in  the  software 
development  specifications,  are  traceable  to 
the  system  specification  and  that  the 
Software  Development  Plan  (SDP)  is 
updated  to  identify  the  required  resources, 
facilities,  personnel,  development  schedule 
and  milestones,  and  software  tools.  The 
developer  may  also  customize  the 
techniques,  methodologies,  standards  and 
procedures  to  be  used  in  software 
development. 

The  outputs  of  the  Software 
Requirements  Analysis  are  final  versions  of 
the  software  specifications,  and  an  updated 
SDP.  These  documents  will  be  reviewed  at 
the  SSR.  The  Computer  Resources  Life 
Cycle  Management  Plan  (CRLCMP)  may 
also  be  updated. 


20.4.2  Preliminary  Design 

After  the  software  allocated  baseline  is 
established,  the  developer  proceeds  into  the 
Software  Preliminary  Design  (Figure  20-4). 
Preliminary  design  activity  determines  the 
overall  structure  of  the  software  to  be  built. 
Based  on  the  requirements,  the  developer 
partitions  the  software  into  components  and 
defines  the  function  of  each  component  and 
the  relationships  between  them.  Input  and 
output  relationships  with  external  devices 
(such  as  displays  and  sensors)  are  refined 
according  to  the  hardware  configuration  and 
software  structure.  The  timing  and  memory 
budget  for  components  are  established  to 
ensure  that  the  software  requirements  can 
be  satisfied  within  the  hardware  constraints. 

The  developer  should  provide  a 
preliminary  design  that  insures  clear 
traceability  of  requirements  from  software 
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Figure  20-4  Software  Preliminary  Deaien 


specifications  down  to  the  software 
components  for  each  CSCI.  The  software 
design  is  reflected  in  the  preliminary 
Software  Design  Document  (SDD)  and 
Interface  Design  Document  (IDD).  These 
documents  will  describe  the  system 
architecture,  memory  and  processing  time 
allocations,  interrupt  requirements,  timing 
and  sequencing  considerations,  and 
input/output  constraint*  for  each  software 
component.  The  developer  should  also 
generate  a  Software  Test  Plan  (STP) 
outlining  the  proposed  test  program  and 
establishing  test  requirements  for  software 
integration  and  testing. 

The  output  of  the  contractor’s  efforts 
are  preliminary  versions  of  the  software 
design  documents  and  the  STP.  These 
documents  are  reviewed  during  the 
Throughout  the  development  effort,  the 


developer  will  conduct  informal  design 
reviews,  inspections,  and  walkthroughs  to 
evaluate  the  progress  and  correctness  of  the 
design  for  each  software  component.  The 
results  of  these  inspections  will  serve  as  the 
basis  for  material  presented  at  the  PDR. 

10.43  Detailed  Design 

The  purpose  of  the  Detailed  Design 
(Figure  20-5)  activity  is  to  logically  define 
and  complete  the  detailed  software  design 
(not  coding)  that  satisfies  the  allocated 
requirements.  The  level  of  detail  of  this 
design  must  be  such  that  the  programming 
of  the  computer  program  can  be 
accomplished  by  someone  other  than  the 
original  designer.  The  component’s  function, 
its  inputs  and  outputs,  plus  any  constraints 
(such  as  memory  size  or  response  time) 
should  be  defined.  Logical,  static,  dynamic 
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relationships  among  the  components  should 
be  specified  and  the  component  and  system 
integration  test  procedures  generated. 

A  complete  detailed  design  includes  not 
only  a  description  of  the  computer  processes 
to  be  performed  but  also  detailed 
descriptions  of  the  data  to  be  processed.  A 
data  dictionary  is  an  effective  way  of 
documenting  this  needed  design  information. 
For  software  that  processes  or  manipulates 
a  large  amount  of  interrelated  data,  the 
structure  of  the  data  itself  should  be  defined. 

Components  coded  in  assembly  language 
or  other  "non-standard"  languages  should  be 
clearly  defined  and  the  reasons  for  the 
departure  justified.  Any  special  conditions 
that  must  be  followed  when  programming 
should  be  similarly  described  and  clearly 
documented  [6].  These  exceptions  are 
normally  addressed  in  the  SDP. 


During  the  entire  design  and 
development  process  the  contractor  should 
document  the  development  of  each  unit, 
component,  and  CSCI  in  software 
development  folders  (SDFs).  A  separate 
SDF  should  be  maintained  for  each  unit, 
each  component,  and  each  CSCI.  The  SDFs 
are  normally  maintained  for  the  duration  of 
the  contract  and  made  available  for 
government  review  upon  request.  A  set  of 
SDFs  may  include  the  following  type  of 
information: 

a.  Design  considerations  and 

constraints 

b.  Design  documentation  and  data 

c.  Schedule  and  status  information 

d.  Test  requirements  and 

responsibilities 


Figure  20-5  Software  Detailed  Design 
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e.  Test  cases,  procedures  and  results. 

The  contractor  documents  and 
implements  procedures  for  establishing  and 
maintaining  SDFs  in  a  Software 
Development  Library.  The  library  is  a 
management  tool  used  by  the  contractor  to 
assist  in  developmental  configuration 
management.  It  serves  as  a  "storage  house" 
to  control  access  of  software,  documentation, 
and  associated  tools  and  procedures  used  to 
facilitate  the  orderly  development  and 

subsequent  support  of  software  [8]. 

A  CDR  is  conducted  at  the  conclusion 
of  the  detailed  design.  The  CDR  should 
assure  that  uie  software  design  satisfies  the 
requirements  of  both  the  system  level 

specification  and  the  software  development 
specifications.  Following  an  acceptable  CDR, 
and  not  before,  the  design  should  be 
released  for  coding  and  unit  testing.  This 

process  may  occur  incrementally  with 

individual  releases  of  CSCIs. 


20.4.4  Coding  and  CSU  Testing 

The  purpose  of  programming  is  to 
translate  the  detailed  software  design  into  a 
programming  language  such  as  Ada.  It  is 
during  the  programming  activity  that  listings 
of  the  source  program  are  generated  (Figure 
20-6).  Based  on  the  detailed  software  design 
presented  in  the  design  specification, 
programming  of  each  unit  is  accomplished 
by  the  assigned  programmer  in  the  specified 
programming  language,  usually  Ada.  As  the 
programming  of  each  unit  is  completed,  the 
programmer  examines  the  program  for 
errors.  Only  after  the  programmer  is 
satisfied  that  the  source  program  correctly 
implements  the  detailed  design,  should  the 
program  be  compiled.  Compiling  translates 
the  source  program  to  its  machine 
executable  form,  the  object  program. 

If  the  detailed  design  is  in  error,  is 
ambiguous,  or  is  not  sufficiently  complete  to 
permit  the  programming  to  continue  without 
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further  definition,  the  programmer  should 
consult  the  original  designer.  The  resolution 
should  be  documented,  and  all  affected 
requirements,  design,  and  test  documentation 
updated  accordingly. 

The  purpose  of  the  unit  testing  activity 
is  to  eliminate  any  errors  that  may  exist  in 
the  units  as  they  are  programmed.  These 
errors  may  be  due  to  mistakes  by  the 
programmer  or  deficiencies  in  the  software 
requirements  and  design  documentation. 
Usually,  the  test  of  a  unit  is  the 
responsibility  of  the  programmer  who 
programmed  the  unit.  Unit  testing  is  the 
activity  that  permits  the  most  control  over 
test  conditions  and  visibility  into  software 
behavior.  An  efficient  software  development 
effort  requires  rigorous  unit  level  test  so  that 
most  errors  are  detected  before  CSC 
Integration  and  Test. 


Besides  producing  the  source  and  object 
code  and  their  respective  listings,  the 
contractor  develops  and  records  in  software 
development  folders  the  informal  test 
procedures  for  each  unit  test  as  well  as  the 
test  results.  The  contractor  will  usually 
conduct  informal  code  inspections  or 
walkthroughs  on  each  coded  unit  and 
component  during  several  stages  of  its 
development.  There  are  no  formal  reviews 
scheduled  during  this  step  of  the 
development  cycle. 

20.4.S  CSC  Integration  and  Testing 

Once  the  software  is  programmed  and 
each  unit  and  component  is  tested  for 
compliance  with  its  design  requirements,  the 
contractor  should  begin  CSC  Integration  and 
Testing  (Figure  20-7).  The  purpose  of  CSC 
Integration  and  Testing  is  to  combine  the 


Figure  20*7  CSC  Integration  and  last 


software  units  and  components  that  have 
been  independently  tested  imo  the  total 
software  product  and  to  demonstrate  that 
this  combination  fulfills  the  system  design. 
The  integration  is  done  in  a  phased  manner 
with  only  a  few  components  being  combined 
at  first,  additional  ones  added  after  the 
initial  combination  has  been  tested,  and  the 
process  repeated  until  all  components  have 
been  integrated.  The  phasing  of  this 
integration  should  be  based  on  the  functional 
capabilities  that  can  be  demonstrated  by 
specific  groups.  There  may  be  some  overlap 
with  the  previous  step  in  that  some  software 
components  may  be  ready  for  integration 
while  others  are  still  being  programmed. 

Most  testing  performed  during  Coding 
and  CSU  Testing,  and  CSC  Integration  and 
Testing  is  called  "informal  testing".  This  term 
doesn’t  imply  that  the  testing  is  "casual"  or 
"haphazard",  but  instead  implies  that  the 
testing  doesn’t  require  government  approval. 


Some  formal  testing  may  be  accomplished 
during  these  steps,  but  most  formal  testing  is 
usually  accomplished  during  the  next  step. 

20.4.6  CSCI  Testing 

After  completion  of  a  successful  TRR, 
the  contractor  will  proceed  with  CSCI 
Testing  (Figure  20-8),  the  last  step  of  the 
software  development  cycle.  The  purpose  is 
to  perform  formal  tests,  in  accordance  with 
the  software  test  plans  and  procedures,  on 
each  CSCI  and  to  establish  the  software 
Product  Baseline.  Testing  during  this  step  is 
intended  to  show  that  the  software  satisfies 
the  Software  Requirements  Specification  and 
the  Interface  Requirements  Specification. 

Throughout  CSCI  testing,  the  contractor 
should  be  updating  all  previous  software 
documentation,  analyzing  test  data, 
generating  the  Software  Test  Reports  (STR), 
and  finalizing  the  Software  Product 


Figure  20-8  CSCI  Testing 
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Specification  (SPS)  (Type  C-5  Specification). 
This  will  be  the  basis  for  the  software 
product  baseline  normally  established  at  the 
PCA,  which  may  immediately  follow,  or  be 
conducted  concurrently  with,  the  FCA  for  a 
software  only  development.  Normally,  the 
PCA  occurs  after  the  software  is  released  for 
integration  and  testing  with  the  system 
following  the  software  FCA  (Figure  20-8). 
During  the  software  FCA  the  government 
verifies  that  the  CSCIs  perform  in 
accordance  with  their  respective 
requirements  and  interface  specifications  by 
examining  the  test  results  and  reviewing  the 
operational  and  support  documentation. 
TTie  PCA  is  the  formal  technical  examination 
of  the  as-built  software  product  against  its 
design,  including  the  product  specification 
and  the  as-coded  documentation. 


The  typical  outputs  of  the  contractor’s 
efforts  in  CSCI  Testing  are  the  1)  Software 
Test  Report  (STR),  2)  operational  and 
support  documentation  such  as  the 
Computer  System  Operator’s  Manual 
(CSOM),  the  Software  Users  Manual 
(SUM),  the  Software  Programmer’s  Manual 
(SPM),  the  Firmware  Support  Manual 
(FSM),  the  Computer  Resources  Integrated 
Support  Document  (CRISD),  the  Version 
Description  Document  (VDD),  and  3)  the 
Software  Product  Specification  (SPS).  Except 
for  updates  and/or  revisions,  all  deliverable 
documentation  should  be  completed  at  this 
time.  Figure  20-9  contains  a  listing  of  the 
standardized  software  documentation,  as 
defined  in  DOD-STD-2167A  [3],  that  may 
be  required  for  software  development 
programs. 


Engineering 


SSS 

SRS 

IRS 

SPS 

SSDO 

VDD 

SDD 

IDD 


Management 


SOP 

SQPP 


CMP 


Operational/ 

Support 


SPM 

SUM 

FSM 

CSOM 

CRISD 


Test 


STP 

STD 

STR 


CRLCMP 


TEMP 


Figure  20-9  Software  Products 
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20.5  SYSTEM  INTEGRATION  AND 

TESTING 

The  purpose  of  System  Integration  and 
Testing  is  to  ensure  that  the  developed 
software  works  with  the  system  in  the 
environment  that  it  was  designed  for  (Figure 
20-10).  The  system  is  turned  over  to  the 
government  after  an  acceptable  Formal 
Qualification  Review  (FOR).  The  FQR  is  a 
system-level  review  that  verifies  that  the 
actual  system  ;  crformance  complies  with  the 
system  requirements.  For  computer 
resources,  it  addresses  the  aspects  of  the 
software  and  hardware  performance  that 
have  been  tested  after  the  FCA  and  PCA. 
A  successful  FQR  is  predicated  on  a 
determination  that  the  system  meets  the 
specified  requirements  in  the  hardware, 
software  and  interface  specifications. 

The  contractor’s  role  will  diminish 
significantly  subsequent  to  the  FQR. 


Contractor  configuration  control  of  the 
software  should  terminate  once  the  product 
baseline  is  approved  and  the  government 
assumes  responsibility.  All  updated 
documentation,  source  and  object  code 
listings,  and  all  other  items  stipulated  in  the 
contract  will  be  delivered  to  the  government. 
The  government  will  then  assume 
configuration  control  responsibility.  The 
contractor,  however,  will  be  available  to 
support  the  government’s  test  and  evaluation 
efforts  and  to  conduct  any  required 
acceptance  tests. 

20.6  TAILORING 

The  purpose  of  tailoring  is  to  reduce  the 
overall  costs  of  an  acquisition,  primarily  by 
reducing  the  amount  and  type  of 
documentation  being  delivered  by  the 
contractor  and  by  eliminating  redundant  or 
unnecessary  testing  or  procedures.  Some 
questions  whose  answers  will  provide 


Figure  20-10  Systea  Integration  and  Test 
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tailoring  guidance  are: 

a.  Is  all  of  the  documentation 
described  in  DOD-STD-  2 167 A  necessary? 

b.  What  documentation  is  already 
available? 

c.  Is  it  cost-effective  to  modify  it? 

d.  Is  the  contractor’s  format 
acceptable? 

e.  How  many  copies  are  actually 
needed? 

f.  How  can  DOD-STD-2167A  be 
tailored? 

g.  Is  a  formal  design  review  necessary 
for  each  CSCI? 

h.  How  should  they  be  scheduled? 


DOD-STD-2167A  [3]  states  that  the 
tailoring  process  for  this  standard  entails  the 
deletion  of  non-applicable  requirements.  But 
how  does  a  program  manager  determine 
which  requirements  are  not  applicable? 
(Figure  20-11  illustrates  the  tailoring 
process). 

Most  tailoring  is  implemented  through 
the  statement  of  work  (SOW).  A  thorough 
understanding  of  requirements  (functional, 
performance,  test,  documentation)  is 
required  in  order  to  properly  tailor  the 
standards  and  specifications. 

The  first  step  is  to  ask  if  the 
requirement  is  appropriate?  If  not,  then 
tailor  it  out  through  the  SOW.  If  the 
requirement  is  appropriate,  then  ask  if  the 
requirement  is  adequate?  If  it  is,  then 
impose  the  requirement  through  the  SOW. 
If  the  requirement  is  not  adequate,  ask  if 
the  requirement'  is  too  restrictive  or  too 


Figure  20-1 1  Tailoring  Process 
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flexible?  If  it’s  too  restrictive,  delete  it  or 
modify  it  in  the  SOW.  If  it’s  toj  flexible, 
add  to  or  modify  the  requirement  in  the 
SOW.  Use  careful  judgement  when  tailoring 
a  program.  Don’t  tailor  areas  arbitrarily 
simply  to  reduce  program  costs  because  in 
the  long  run,  this  may  increase  life  cycle 
costs. 

20.7  ENGINEERING  STUDIES 

Systems  engineering  studies  are  based 
on  the  concept  of  a  hierarchy  of 
requirements  starting  with  system  level 
requirements  and  ending  with  detailed 
engineering  specifications  and  data.  System 
definition  proceeds  by  refining  each  level  of 
requirement  into  subordinate  requirements 
until  the  entire  system  is  described. 
Computer  resources  are  considered  as  an 
integral  part  of  the  system  and  are  subject  to 
tradeoff  and  optimization  studies.  Systems 
engineering  studies  will  normally  include: 

a.  Requirements  Definition 
Requirements  definitions  begins  with  a 
preliminary  allocation  of  requirements  to 
either  hardware  or  software.  The 
requirements  for  each  software  configuration 
item  are  documented  in  a  Software 
Requirements  Specification  (SRS).  The  SRS 
is  authenticated  at  the  Software  Specification 
Review  (SSR)  which  is  normally  held  during 
the  FSD  phase. 

b.  Interface  Definition  -  The 
Computer  Resources  Working  Group 
(CRWG),  in  conjunction  with  the  Interface 
Control  Working  Group,  addresses  system 
and  subsystem  interface  requirements  that 
may  affect  computer  resources.  The 
requirements  for  interfaces  between 


Computer  Software  Configuration  Items 
(CSCIs)  and  other  system  configuration 
items  are  documented  in  one  or  more 
Interface  Requirements  Specifications  (IRS) 
that  are  also  authenticated  at  the  SSR.  The 
SRS  and  the  IRS  form  the  CSCI  allocated 
baseline. 

c.  Tradeoff  and  Optimization  - 

Tradeoff  and  optimization  studies  should 
consider  such  issues  as: 

1)  Tradeoffs  between  computer 
software  and  computer  hardware 

2)  Required  computer  processor 
architectural  features  such  as  memory  size, 
processor  speed,  input  and  output  capacity, 
and  spare  capacity 

3)  Use  of  standard  equipment, 
higher  order  languages,  instruction  set 
architectures,  and  interfaces 

4)  Alternate  approaches  for  meeting 
system  security  requirements 

5)  Improved  supportability  versus 
improved  performance 

6)  Use  of  existing  government 
resources  or  commercial  off-the-shelf 
resources  versus  new  development. 

d.  Feasibility  Studies  -  These  studies 
determine  the  feasibility  of  alternative 
allocations  of  system  requirements  to 
computer  resources  and  the  derivation  of 
data  for  formulating  budgets  and  schedules. 

e.  Risk  Analysis  -  The  program  office 
must  identify  the  major  risks  to  the  software 
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development  effort  and  insure  that  plans  for 
managing  these  risks  are  incorporated  into 
the  system  level  risk  management  plan. 

f.  Software  Support  Studies  - 
Software  support  studies  are  conducted  to 
refine  the  system  support  concept  and  to 
allocate  software  support  requirements. 
These  studies  determine  how  operational 
system  software  will  be  identified.  Two 
potential  methods  are  self-identification  of 
executing  software  and  identification  plates 
affixed  to  the  outside  of  the  computer. 

20.8  SUMMARY 

Software  that  is  part  of  a  weapon 
system  is  managed  by  partitioning  into 
CSCIs.  Each  CSCI  is  managed  individually 
and  follows  its  own  software  development 
cycle.  Software  development  activities  can 
be  broken  down  into  six  steps;  any  of  which 
can  be  repeated  as  many  times  as  recessary 
during  the  development  cycle.  These  six 
steps  are  Software  Requirements  Analysis, 
Preliminary  Design,  Detailed  Design,  Coding 
and  CSU  Testing,  CSC  Integration  and 


Testing,  and  CSCI  Testing.  These  steps 
typically  occur  during  the  Full  Scale 
Development  Phase. 

DOD-STD-2167A  [3]  is  the  approved 
standard  to  be  used  by  DOD  agencies  for 
software  development.  It  is  to  be  used  in 
conjunction  with  DOD-STD-2168  [9].  These 
two  standards  are  not  intended  to  discourage 
the  use  of  any  particular  software 
development  method,  but  instead,  aid  the 
software  manager  in  developing  and 
maintaining  quality  software.  They  should  be 
used  throughout  the  acquisition  life  cycle  and 
tailored  according  to  system  needs. 

It  is  especially  important  to  develop  the 
product  as  a  system.  Never  lose  sight  of  the 
fact  that  hardware  and  software  development 
are  intimately  related.  Although  they  are 
developed  in  parallel,  software  is  almost 
always  in  the  critical  path  and  it  is  up  to  the 
systems  engineer  to  insure  proper  integration 
of  the  two  through  carefully  planned  reviews 
and  audits.  The  talents  of  an  independent 
verification  and  validation  (IV&V)  activity 
may  be  used  to  aid  in  this  process. 
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CHAPTER  21 


COMPUTER  AIDED  TECHNICAL  MANAGEMENT 


21.1  INTRODUCTION 

It  is  fairly  accurate  to  assume  that  use  of 
computers  and  chip-based  processors  will 
soon  automate  a  majority  of  the  traditional 
" technical  management "  functions;  thereby, 
increasing  productivity,  efficiency,  and 
effectiveness  of  the  engineering  community. 
The  Department  of  Defense  (DoD)  is 
encouraging  the  application  of  this  capability 
through  a  variety  of  incentives  and  funded 
programs;  e.g.,  computer  aided  design 
(CAD),  computer  aided  manufacturing 
(CAM),  and  computer  aided  acquisition  and 
logistics  support  (CALS).  Industry  has  led 
the  way  in  CAD  and  CAM  by  linking 
together  the  requirements  and  capabilities  of 
them  through  a  common  data  base.  DoD  is 
attempting  to  integrate  the  requirements  of 
CALS  into  the  CAD/CAM  structure  in  order 
to  achieve  an  integrated  computer  aided 
technical  management  (CATM)  system  as 
reflected  in  Figure  21-1.  It  is  anticipated  that 
the  improvements  of  CATM  to  the 
acquisition  process  will  evolve  in  the  same 
manner  that  improvements  to  many 
development,  production,  and  integration 
processes  have  evolved  through  successful 
automation.  The  " obvious "  direct  results  of 
automation  will  naturally  be  improvement  in 
both  1)  the  quality  of  the  products,  services, 
and  information  that  result  from  these 
processes  and  2)  the  reduction  of  the 
resources  required  to  execute  the  processes 
and  functions. 


In  the  " 'factory  of  the  future "  it  is 
prophesied  that  a  single,  integrated, 
multi-purpose  data  base  will  be  developed  to 
design,  produce,  and  provide  necessary 
information  for  the  technical  management  of 
a  product  or  service  acquisition  program. 
Furthermore,  it  is  believed  that  automatic 
production  machine  cells  will  transform  raw 
materials  into  finished  products  without 
human  effort  or  intervention.  Changing  from 
one  product  to  another  will  require  only  a 
change  V  system  software.  System  support 
will  be  more  efficient  and  effective  because 
a  common  data  base  will  be  used  for 
analysis,  support  system  design,  configuration 
control,  maintenance  data  interpretation,  and 
supply  support. 

To  date,  after  about  thirty  years  of  study 
and  research,  the  most  optimistic  extents  of 
the " power  of  automation "  have  proven  elusive 
because  of  the  usual  kinds  of  problems 
encountered  in  transition  of  any  new 
technology  from  the  laboratory  to  the  work 
place.  The  following  are  some  examples  of 
these  problems: 

a.  The  high  initial  cost  of  automation 
technology  increases  capital  investment 
payback  periods  and  makes  most  automation 
projects  unattractive  for  "bottom  line”,  short 
term  profit  oriented  managers  and 
enterprises.  Because  of  these  high  initial 
costs,  automation  is  perceived  to  be  a 
significant  business  risk  and  a  potential  cost 


21-1 


COMPUTER  AIDED  TECHNICAL  MANAGEMENT 


21-2 


effectiveness  problem;  especially,  if  the 
automation  system  is  poorly  designed, 
inflexible,  or  not  "user  friendly". 

b.  Expertise  in  certain  engineering 
technical  functions  and  processes  has  not 
proven  conducive  to  transition  from  people 
to  computers.  The  requirements  of  the 
" technical  arts ",  which  include  the  knowledge 
and  experience  of  activities  carried  out  in 
the  transition  from  design  to  production  and 
in  correction  of  product  defects  by  integrated 
product  design  and  manufacturing  process 
changes,  will  be  difficult  to  translate  into  the 
requirements  by  which  i  lachines  can  execute 
simply  or  reliably. 

c.  Vision  based  adaptive  production  line 
activities  (which  includes  identification, 
manipulation,  assembly,  inspection,  and 
adjustment  of  parts  and  components  into 
finished  complex  products)  are  extremely 
difficult  to  analyze  and  to  breakdown  into 
sets  of  machine  executable  instructions  and 
routines.  It  is  highly  unlikely,  from  a  cost 
effective  perspective,  that  robots  will  replace 
the  worker  force  in  the  near  future.  The 
flexibility  and  reasoning  power  of  the  human 
cannot  yet  be  achieved  through  the 
automation  process. 

d.  Development  of  integrated  systems, 
capable  of  many  different  technical 
management,  engineering,  production,  and 
logistics  functions  (using  a  single  integrated 
data  base),  can  be  very  expensive  in  terms 
of  resource  investments.  Well  designed  and 
innovative  single  purpose  automated  systems 
are  available  while  multiple  use  systems  have 
not  effectively  transitioned  out  of  the  "design 
evaluation"  stage;  therefore,  making  it 
difficult  to  achieve  an  optimum  application 


of  automi  ition  technology. 

21.2  COMPUTER  AIDED  TECHNICAL 
MANAGEMENT  (CATM) 

Despite  the  problems  mentioned,  which 
have  pus  hed  the  concepts  of  the  "workerless 
factory"  and  the  " engineer  in  the  black  box " 
away  from  the  near  horizon,  automation  of 
many  technical  management  functions  and 
engineering  processes  have  been  carried  out 
with  very  encouraging  results. 

The  hirst  computer  applications  which  still 
provide  the  greatest  productivity 
improvements  and  returns  on  investment 
with  computer  automation  are  those 
requiring  collection,  storage,  and 
manipulation  of  numerical  and  record  type 
information.  Simply  stated,  any  technical 
management,  engineering,  or  production 
process;  which  presently  is  based  on  paper 
passing,  formula  and  number  crunching,  or 
organization  and  control  will  be  improved  in 
terms  of  speed,  accuracy,  and  flexibility  by 
automation.  In  most  cases  powerful  software 
systems  are  available  and  have  been 
effectively  applied  to  technical  management, 
engineering,  and  problem  solving  tasks. 

21.2.1  Management  Information  Systems 
(MIS) 

Resource  usage  reporting,  scheduling, 
status  accounting,  and  other  reporting  and 
recording  functions  are  historically  known 
and  used  as  MAS  systems.  Also  included  in 
this  category  a/;e  so-called  decision  support 
systems  (DSS)  which  utilize  experiential 
algorithms  to  1)  evaluate  program 
management  status  data  and  2)  recommend 
potential  corrective  actions  if  required. 
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These  programs  are  effectively  appli  ed  to  1) 
eliminate  paper  passing  between  the 
contractor  and  the  government,  2)  reduce 
delays  in  change  analyses  and  appro  vals,  3) 
implement  real  time  configuration 
management  (CM),  4)  track  technical 

performance  measurement  ('TPM) 
parameters,  5)  evaluate  logistic  support 
analysis  (LSA),  and  6)  provide  a  readily 
accessible,  usable,  and  easily  stored  p  rogram 
history. 

21.2.2  Computer  Aided  Engineering  (  CAE) 

Computer  aided  engineering  (C>\£)  is 
composed  of  numerous  engineering  design 
and  analysis  programs,  collectively  known  as 
CAD,  CAME/CME  (Computer  Aided 
Mechanical  Engineering),  and  'CASE 
(Computer  Aided  Systems  Engineering), 
which  have  been  successfully  developed  and 
widely  utilized.  These  programs;  1) 
incorporate  a  computer  graphics  capability 
to  replace  the  manual  hand  drawing 
practice,  and  2)  create  data  bases  of 
standard  components,  previously  designed 
parts,  and  standard  drawing  practices  and 
conventions  to  allow  the  engineer  to  pro  duce 
large  quantities  of  quality  drawings  and  other 
documentation  in  short  periods  of  time.  The 
CASE  programs  have  analagous  functions  in 
the  design  of  software  systems  using  existing 
modules  and  design  standards.  Theoret  ical 
evaluation  and  analysis  of  proposed  desi  gns 
in  such  areas  as  heat  transfer,  electrical  nnd 
anacoic  properties  and  stress  reaction  can  be 
carried  out  by  including  inter  active  "look  up" 
tables  and  standard  formulae  routines. 
Marginal  designs  can  be  eliminated  without 
the  expense  and  delay  of  fabrication  and 
test.  Test  and  evaluation  data  and  report 


generation,  manipulation,  and  storage  is 
greatly  improved  by  the  use  of  automated 
collection  systems.  Testing  can  be 
significantly  expedited  and  organized  through 
the  use  of  programmed  testing  routines  if 
the  program  size  can  support  the  additional 
expense  of  development.  The  cycle  of  test, 
analyze,  and  fix  (TAAF)  can  be  similarly 
effected  by  an  automated  program’s  speed, 
analysis  power,  and  rapid  design  change 
potential. 

21.23  Computer  Aided  Manufacturing 
(CAM) 

CAM  and  Robotics  arc  in  use  in  many 
different  types  of  production  and 
manufacturing  facilities.  Evolution  of  the 
Numerical  Control  (NC)  tape  controlled 
machines  into  Computer  Integrated 
Machining  (CIM)  cells  has  taken  place  in 
some  organizations  (raw  materials  are 
converted  into  completed  products  without 
human  intervention).  Experience  with  robots 
and  CIM  has  produced  some  interesting 
conclusions  and  rules  for  effective  use  of 
manufacturing  automation.  Robots  have 
been  found  to  be  best  applied  in  simple, 
repetitive,  high  volume  tasks  such  as  "put 
and  place "  (e.g., insertion  of  electrical 

components  onto  circuit  boards),  especially 
when  the  products  are  designed  or 
redesigned  for  optimum  automatic  assembly. 

Both  robots  and  CAM  are  very  expensive 
to  acquire  and  are  cost  effectively  applied 
oidy  to  highly  standardized  processes  and 
products.  One  typical  example  of  tills 
general  rule  is  in  machining  of  very  high 
value  intricate  parts,  such  as  turbine  engine 
cases  which  are  now  almost  exclusively  done 
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by  computer  controlled  machines. 

Many  integrated  CAD/CAM/CIM 
systems,  in  which  the  CAD  data  base  is 
interactively  used  by  the  CAM/CIM  function, 
are  being  effectively  applied  in  both 
government  and  industry.  The  Navy  is 
contracting  for  the  development  of  a 
multipurpose  CAD/CAM  system  to  be  used 
for  development  of  inclusive  technical  data 
bases  on  major  systems  by  all  the  service 
commodity  commands.  Compatibility  of  this 
unified  government  system  with  the  different 
systems  in  use  by  major  contractors  will  be 
a  major  issue  in  future  acquisitions. 

The  general  rules  regarding  robotics  and 
CAD/CAM/CIM  application  are  similar  to 
those  regarding  MIS  systems.  In  the  areas  of 
data  management,  collection,  storage,  and 
manipulation,  automatic  systems  are 
decisively  more  productive  and  effective  than 
human  effort  and  hard  copy  documentation 
and  should  be  utilized  wherever  possible. 

Highly  standardized  manufacturing  of 
low  change  products  in  large  numbers  is  also 
becoming  an  area  with  excellent  automation 
payback  potential.  It  should  be  noted  that 
even  in  these  cases  that  redesign  of  the 
product  or  the  manufacturing  process  for 
efficient  automation  may  be  required  and 
should  always  be  carried  out  as  a  final  check 
before  implementation. 

As  applications  proceed  toward  intricate 
and  difficult  multi-axis  copying  of  human 
motions  and  adaptive  activities  such  as 
assembly  and  adjustment  of  complicated 
mechanisms,  the  payback  on  investment  in 
robotics  and  computer  controlled 
manufacturing  rapidly  diminishes.  The  sunk 


costs  and  time  delay  of  redesign  for 
automated  manufacture  can  be  avoided  if 
fiscal  and  budget  stability  and  product 
standardization  can  be  established  early 
enough  in  the  program  to  allow  automated 
manufacturing  process  design  to  proceed  in 
parallel  with  product  design  and 
development.  This  parallel  product  and 
process  design  is  encouraged  as  a  matter  of 
policy  by  DOD  and  is  descriptively  called 
"concurrent  engineering".  State  of  the  art 
systems  design,  typical  of  DOD  acquisitions, 
and  insertion  of  the  latest  hardware  and 
software  technology  should  be  attempted 
only  by  this  method. 

21.2.4  Computer  Aided  Acquisition  and 
Logistics  Support  (CALS) 

The  most  recently  proposed  automation 
opportunity  of  .interest  to  the  program 
management  office  (PMO)  is  CALS.  CALS 
is  a  DOD  and  industry  supported  initiative 
which  will  enable  and  accelerate  the 
integration  and  use  of  digital  technical 
information  development,  manipulation,  and 
transfer  in  the  management  and  functional 
efforts  of  weapon  system  design,  evaluation, 
manufacture,  and  support.  DOD  initiated 
discussions  of  CALS  policy  and  techniques  in 
1985  in  an  attempt  to  solve  the  rapidly 
growing  problem  and  expense  of  unique  and 
incompatible  automated  systems  used  by  the 
government  and  industry  in  weapon  system 
acquisition  and  technical  management.  CALS 
will  require  government  organizations  and 
their  contractors  to  adopt  specific 
information  exchange  standards  and  to 
integrate  their  technical  and  support 
information  databases.  The  ultimate  goal  of 
CALS  are  to  allow  the  government  to  send, 
receive,  distribute,  and  use  technical 
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information  in  electronic  form.  At  the 
present  time  the  Under  Secretary  of  Defense 
(Acquisition)  has  requested  industry  to 
accept  the  lead  in  the  development  of 
appropriate  standards  and  efforts  to  meet 
the  stated  CALS  goals.  The  major  difficulties 
at  this  point  are  to  settle  on  an  agreeable 
standard  and  to  build  confidence  and 
experience  in  use  of  automatic  data  transfer. 
The  next  round  of  problems  will  include 
budgeting,  system  certification  and 
maintenance,  ,  overcoming  unwillingness  to 
have  real  time  data  transfer  and  problem 
exposure,  and  worker  inertia  in  learning  and 
using  the  system  near  its  potential.  More 
information  on  CALS  can  be  found  in 
Chapter  19  of  this  guide. 

213  SUMMARY 

In  evaluating  or  planning  applications  of 
CATM  to  new  acquisition  programs,  the 
systems  engineer  must  balance  common 
sense  and  practical  judgement  against  the 
estimates  of  resource  costs  and  returns  on 
investment.  At  the  present  time  there  are  no 
DOD  or  major  industry  wide  standard 
systems  or  system  design  standards  for  CAD, 
CAM,  CASE,  CIM,  CALS,  or  MIS 
(advertised  as  "program  management" 
planning,  scheduling,  and  reporting)  systems; 
however,  use  of  initiative  and  innovation  for 
automatic  digital  information  transfer, 
storage,  and  manipulation,  between  the 
government  and  contractor,  is  both  possible 
and  potentially  very  productive.  Contractor 
format  data  and  on-line  transfer  should  be 


evaluated,  as  a  minimum.  Investment  in  a 
mutually  acceptable  system  and  format  has 
been  carried  out  by  many  recent  major 
DOD  and  System  Commands  programs. 
DOD  policy  on  electronic  acquisition  data 
transfer  and  system  standards  should  be 
forthcoming  in  the  near  future.  In  the 
interim,  the  benefits  of  automation  should 
justify  resource  costs  to  most  major 
programs,  with  the  obvious  exception  of 
non  development  or  firm  fixed  price 
procurements  with  minimum  data  transfer 
requirements. 

The  big  picture  problem  for  this 
computer  aided  revolution  is  already  well 
documented  and  accepted-  lack  of 
technically  skilled  workers.  A  long  term 
solution  is  already  in  being-  a  new 
generation  of  workers  being  raised  in  the 
computer  aided  environment.  Computer 
aided  design  and  manufacturing  are  here  to 
stay.  They  are  meely  the  first  steps.  Among 
the  tasks  remaining  to  reach  the  mature 
integrated  system  are  defining  the  input- 
output  relationships  between  the  engineering 
activities  (e.g,  computer  aided  technical 
performance  measurement  and  computer 
aided  cost  measurement)  and  engineering 
specialties  or  disciplines.  The  advancing 
computer  and  data  systems  technologies 
afford  the  means  to  move  toward  a  truly 
integrated  system;  but,  without  a  cultural 
change  in  the  way  the  functional  disciplines 
interrelate  and  conduct  their  activities,  true 
progress  will  be  impeded  [1]. 
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ACAT-  Acquisition  Category  (Navy) 

ACSN-  Advance  Change/Study  Notice 
ADM-  Advanced  Development  Model 
ALCM-  Air  Launched  Cruise  Missile 
BIT-  Built-In  'Zest 
BITE-  Built-In  Test  Equipment 
CCB-  Configuration  Control  Board 
CDR-  Critical  Design  Review 
CDRL-  Contract  Data  Requirements  List 
CDS-  Concept  Description  Sheet 
C/E-  Concept  Exploration/  Definition 
Cf  Consequence  of  Failure 
Cl-  Configuration  Item 
CM-  Configuration  Management 
CPM-  Cost  Performance  Measurement 
CPR-  Cost  Performance  Report 

CRISD-  Computer  Resources  Integrated  Support  Docum  ent 
CRLCMP-Computer  Resources  Life  Cycle  Management  Plan 
CSC-  Computer  Software  Component 
CSCI-  Computer  Software  Configuration  Item 
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C/SC S-  Cost/Schedule  Control  System 

C/SCSC-Cost/Schedule  Control  System  Criteria 

CSOM-  Computer  Syst  em  Operator’s  Manual 

CSSR-  Cost  Schedule  5  s  tatus  Report 

CWBS-  Contract  Work  Breakdown  Structure 

DAB-  Defense  Acqu  ivition  Board 

DCP-  Decision  Coordinating  Paper 

DID-  Data  Item  Description 

DoD-  Department  of  Defense 

DOT&E-  Director  C  )perational  Test  and  Evaluation 

DS-  Design  Shee  ; 

DSMC-  Defense  S  ystems  Management  College 

DSSP-  Defense  St  andardization  and  Specification  Program 

DT-  Developme  ni:  Testing 

DTC-  Design  to  Cost 

DT&E-  Develop  iment  Test  and  Evaluation 

DUSDRE-Depu  cy  Under  Secretary  of  Defense  for  Research  and  Engineering 
D/V-  Concept  Demonstration/Validation 
ECP-  Engine*  :ring  Change  Proposal 

ECR-  Embed  ded  Computer  Resources,  Engineering  Change  Request 
EDM-  Engineering  Development  Model 
EIMS-  End  I  te  rn  Maintenance  Sheet 
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FCA-  Functional  Configuration  Audit 

FFBD-  Functional  Flow  Block  Diagram 

FIS-  Facility  Interface  Sheet 

FQR-  Formal  Qualification  Review 

FSD-  Full  Scale  Development 

GFE-  Government-Furnished  Equipment 

HWCI-  Hardware  Configuration  Item 

ICD-  Interface  Control  Document 

ICWG-  Interface  Control  Working  Group 

ILS-  Integrated  Logistic  Support 

ILSP-  Integra;  ed  Logistic  Support  Plan 

IMIP-  Industrial  Modernization  Incentives  Program 

IOC-  Initial  Operating  Capability 

IPS-  Integrated  Program  Summary 

IRA-  Industrial  Resource  Analysis 

IRN-  Interface  Revision  Notice 

IRS-  Interface  Requirements  Specification 

LCC-  Life  Cycle  Cost 

LCCP-  Life  Cycle  Cost  Plan 

LLCSC-  Lower-Level  Computer  Software  Components 
LREP-  I  jaw  Rate  Initial  Production 
LRU-  Line  Replaceable  Unit 
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LSA-  Logistic  Support  Analysis 

LSAR-  Logistic  Support  Analysis  Record 

MCCR-  Mission-Critical  Computer  Resources 

MCCS-  Mission-Critical  Computer  System 

MM/CC-  Milestone  Measurement/Cost  Correlation 

MTBF-  Mean  Time  Between  Failure 

MTBM-  Mean  Time  Between  Maintenance 

MTTR-  Mean  Time  To  Repair 

NATO-  North  Atlantic  Treaty  Organization 

OCD-  Operational  Concept  Document 

OFPP-  Office  of  Federal  Procurement  Policy 

O&M*  Operation  and  Maintenance 

OMB-  Office  of  Management  and  Budget 

O&S-  Operating  and  Support,  Operation  and  Support 

OSD-  Office  of  the  Secretary  of  Defense 

OT-  Operational  Testing 

OTA-  Operational  Test  Agency 

OT&E-  Operational  Test  and  Evaluation 

PAE-  Performance  Achievement  Event 

PCA-  Physical  Configuration  Audit 

PDM-  Program  Decision  Memorandum 

PDR-  Preliminary  Design  Review 
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PEP-  Producibility  Engineering  and  T  <anning 
Py-  Probability  of  Failure 
PI-  Product  Improvement 
P^I-  Pre-Planned  Product  Improvement 
PIRN-  Preliminary  Interface  Revision  Notice 
POM-  Program  Objectives  Memorandum 

PRICE-  Programmed  Review  of  Information  for  Costing  and  Evaluation 

PRR-  Production  Readiness  Review 

PS-  Production  Sheet 

RAS-  Requirements  Allocation  Sheet 

R&D-  Research  and  Development 

RDT&E-  Research,  Development,  Test  and  Evaluation 

RFP-  Request  for  Proposal 

R&M-  Reliability  and  Maintainability 

RAM-  Reliability, Availability, Maintainability 

SBD-  Schematic  Block  Diagram 

SCN-  Specification  Change  Notice 

SCP-  System  Concept  Paper 

SDR-  System  Design  Review 

SE-  Systems  Engineering 

SECDEF-Secretary  of  Defense 

SEMP-  Systems  Engineering  Management  Plan 
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SOW-  Statement  of  Work 


SQEP-  Software  Quality  Evaluation  Plan 

SRR-  System  Requirements  Review 

SRS-  Software  Requirements  Specification 

SSR-  Software  Specification  Review 

STR-  Software  Test  Report 

SUM-  Software  User’s  Manual 

TDRS-  Tracking  and  Data  Relay  Satellite 

T&E-  Test  and  Evaluation 

TEMP-  Test  and  Evaluation  Master  Plan 

TLCSC-  Top-Level  Computer  Software  Component 

TLS-  Time  Line  Sheet 

TPM-  Technical  Performance  Measurement 

TRACE-  Total  Risk  Assessing  Cost  Estimates 

TRR-  Test  Requirements  Review 

TRS-  Test  Requirements  Sheet 

TSR-  Trade  Study  Report 

WBS-  Work  Breakdown  Structure 
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